Keep Track of Temporary Locations for Uploaded Files Mvc

Introduction

Sending large files to an MVC/Web-API Server can be problematic. This commodity is almost an alternative. The arroyo used is to intermission a large file up into small-scale chunks, upload them, then merge them back together on the Server via file transfer by partitioning. The article shows how to send files to an MVC Server from both a webpage using JavaScript, and a Spider web-grade httpClient, and tin be implemented using either MVC or Web API.

In my experience, the larger the file you lot need to upload to a website/API, the bigger the potential problems you encounter. Even when you put the correct settings in place, adjust your web.config, make certain you lot use the right multiplier for maxRequestLength and maxAllowedContentLength and of course don't forget about executionTimeout (eek!), things can still go wrong. Connections can neglect when the file is *near* transferred, servers unexpectedly (Murphy'south constabulary) run out of space, etc., the list goes on. The diagram below demonstrates the basic concept discussed in this article.

Background

The concept for this solution is very simple. The attached code works (I have it started in production), and can exist improved by you in many ways. For example, for the purposes of this commodity the original large file is broken into app. 1mb chunks, and uploaded to the server sequentially, i clamper at a time. This could, for example, be fabricated more efficient by threading, and sending chunks in parallel. Information technology could as well be made more robust by calculation fault tolerance, auto-resume into a residual-api architecture etc. I get out y'all to implement these features yourself if yous need them.

The code consists of ii parts - the initial file-carve up/division into chunks, and the final merge of the chunks back into the original file. I volition demonstrate the file-separate using both C# in a spider web-form, and JavaScript, and the file-merge using C# server-side.

File split

The concept of splitting a file is very bones. We transverse the file in a binary stream, from position zero, upwardly to the concluding byte in the file, copying out chunks of binary information along the way and transferring these. More often than not nosotros ready an arbitrary (or advisedly thought out!) chunk size to extract, and apply this as the amount of data to take at a time. Anything left over at the cease is the final chunk.

In the example beneath, a clamper size of 128b is set. For the file shown, this gives us iii 10 128b chunks, and i ten 32b. In this example there are 4 file chunks resulting from the split and to transfer to the server.

C# File Carve up

The accompanying demo "WinFileUpload" is a simple Windows forms application. Its sole role is to demonstrate splitting a sample large file (l MB) in C#, and using a HTTPClient to mail the file to a web-server (in this case, an MVC Server).

For this C# case, I have a course called Utils  that takes some input variables such as maximum file chunk size, temporary folder location, and the proper noun of the file to split. To carve up the file into chunks, we call the method "SplitFile". SplitFile works its mode through the input file and breaks it into dissever file chunks. We so upload each file chunk it using "UploadFile".

1. Utils ut = new  Utils();
2. ut.FileName ="hs-2004-15-b-full_tif.bmp" ;
3. ut.TempFolder = Path.Combine(CurrentFolder,"Temp" );
4. ut.MaxFileSizeMB = i;
5. ut.SplitFile();
7. foreach  ( string  File in  ut.FileParts)
8.   {
9.     UploadFile(File);
10.   }
11. MessageBox.Testify("Upload complete!" );

The file upload method takes an input file-name, and uses a HTTPClient to upload the file. Notation the fact that we are sending MultiPartFormData to bear the payload.

1. public bool  UploadFile( string  FileName)
2. {
3. bool  rslt = false ;
4. using  (var client = new  HttpClient())
5.     {
6. using  (var content = new  MultipartFormDataContent())
7.         {
8.          var fileContent =new    ByteArrayContent(System.IO.File.ReadAllBytes(FileName));
9.          fileContent.Headers.ContentDisposition =new
10.              ContentDispositionHeaderValue("zipper" )
11.                {
12.                 FileName = Path.GetFileName(FileName)
13.                };
14.          content.Add(fileContent);
16.         var requestUri ="http://localhost:8170/Home/UploadFile/" ;
17. endeavor
18.             {
19.                 var result = client.PostAsync(requestUri, content).Result;
20.                 rslt =true ;
21.             }
22. catch  (Exception ex)
23.             {
25.                 rslt =false ;
26.             }
27.         }
28.     }
29. return  rslt;
30. }

And so, that'due south the supporting code out of the way. One of the critical things to be aware of adjacent is the file naming convention that is being used. It consists of the original file-name, plus a code-parsable tail "_part." that volition be used server-side to merge the different file chunks dorsum into a single contiguous file again. This is but the convention I put together - yous can modify it to your own requirements, but exist sure you are consistent with it.

The convention for this example is,

Name = original proper name + ".part_N.X" (N = file part number, X = total files).

Here is an instance of a picture file split into three parts.

1. MyPictureFile.jpg.part_1.3
2. MyPictureFile.jpg.part_2.3
3. MyPictureFile.jpg.part_3.3

Information technology doesn't matter what order the file chunks are sent to the Server. The important affair is that some convention, like the in a higher place is used, and then that the Server knows (a) what file role it is dealing with and (b) when all parts have been received and can be merged back into one big original file once again.

Side by side, here is the meat of the C# code that scans the file, creating multiple clamper files ready to transfer.

1. public bool  SplitFile()
2. {
3. bool  rslt = false ;
4. string  BaseFileName = Path.GetFileName(FileName);
6. int  BufferChunkSize = MaxFileSizeMB * (1024 * 1024);
8. const int  READBUFFER_SIZE = 1024;
9. byte [] FSBuffer = new byte [READBUFFER_SIZE];
11. using  (FileStream FS = new  FileStream(FileName, FileMode.Open, FileAccess.Read, FileShare.Read))
12.     {
14. int  TotalFileParts = 0;
15. if  (FS.Length < BufferChunkSize)
16.         {
17.             TotalFileParts = 1;
18.         }
19. else
20.         {
21. bladder  PreciseFileParts = (( float )FS.Length / ( float )BufferChunkSize);
22.             TotalFileParts = (int )Math.Ceiling(PreciseFileParts);
23.         }
25. int  FilePartCount = 0;
27. while  (FS.Position < FS.Length)
28.         {
29. cord  FilePartName = String.Format( "{0}.part_{1}.{2}" ,
30.             BaseFileName, (FilePartCount + ane).ToString(), TotalFileParts.ToString());
31.             FilePartName = Path.Combine(TempFolder, FilePartName);
32.             FileParts.Add(FilePartName);
33. using  (FileStream FilePart = new  FileStream(FilePartName, FileMode.Create))
34.             {
35. int  bytesRemaining = BufferChunkSize;
36. int  bytesRead = 0;
37. while  (bytesRemaining > 0 && (bytesRead = FS.Read(FSBuffer, 0,
38.                  Math.Min(bytesRemaining, READBUFFER_SIZE))) > 0)
39.                 {
40.                     FilePart.Write(FSBuffer, 0, bytesRead);
41.                     bytesRemaining -= bytesRead;
42.                 }
43.             }
45.           FilePartCount++;
46.         }
48.     }
49. return  rslt;
50. }

That's it for the C# client-side - nosotros volition see the result and how to handle things server-side later in the article. Adjacent, let'southward expect at how to do the same thing in Javascript, from a web-browser.

JavaScript File Separate

NB - The JavaScript code, and the C# Merge code are contained in the fastened demo file "MVCServer"

In our browser, we have an input control of type "file", and a button to call a method that initiates the file-split and data transfer.

1. < input blazon = "file" id = "uploadFile" proper noun = "file" /> < a class = "btn btn-principal" href = "#" id = "btnUpload" > Upload file </ a >

On document ready, we bind to the click outcome of the button to phone call the main method.

1. $(certificate).ready( function  () {
2.     $('#btnUpload' ).click( role  () {
3.         UploadFile($('#uploadFile' )[0].files);
4.         }
5.     )
6. });

Our UploadFile method does the work of splitting the file into chunks, and as in our C# example, passing the chunks off to some other method for transfer. The primary difference here is that in C#, we created individual files, in our JavaScript example, we are taking the chunks from an assortment instead.

1. function  UploadFile(TargetFile)
2. {
4. var  FileChunk = [];
6. var  file = TargetFile[0];
8. var  MaxFileSizeMB = 1;
9. var  BufferChunkSize = MaxFileSizeMB * (1024 * 1024);
10. var  ReadBuffer_Size = 1024;
11. var  FileStreamPos = 0;
13. var  EndPos = BufferChunkSize;
14. var  Size = file.size;
17. while  (FileStreamPos < Size)
18.     {
20.         FileChunk.push(file.piece(FileStreamPos, EndPos));
21.         FileStreamPos = EndPos;
22.         EndPos = FileStreamPos + BufferChunkSize;
23.     }
25. var  TotalParts = FileChunk.length;
26. var  PartCount = 0;
28. while  (clamper = FileChunk.shift())
29.     {
30.         PartCount++;
32. var  FilePartName = file.name + ".part_"  + PartCount + "."  + TotalParts;
34.         UploadFileChunk(chunk, FilePartName);
35.     }
36. }

The UploadFileChunk takes the part of the file handed by the previous method, and posts it to the Server in a similar way to the C# example.

1. function UploadFileChunk(Clamper, FileName)
2. {
3.     var FD =new  FormData();
4.     FD.suspend('file' , Chunk, FileName);
5.     $.ajax({
6.         type:"Postal service" ,
7.         url:'http://localhost:8170/Home/UploadFile/' ,
8.         contentType:false ,
9.         processData:false ,
10.         data: FD
11.     });
12. }

File merge

NB -The JavaScript lawmaking, and the C# Merge code are contained in the attached demo file "MVCServer"

Over on the Server, exist that MVC or Web-API, nosotros receive the private file chunks and demand to merge them back together again into the original file.

The first thing we do is put a standard POST handler in identify to receive the file chunks being posted upwardly to the Server. This code takes the input stream, and saves it to a temp folder using the file-name created by the client (C# or JavaScript). Once the file is saved, the lawmaking so calls the "MergeFile" method which checks if it has plenty file chunks available yet to merge the file together. Note that this is simply the method I have used for this article. You may decide to handle the merge trigger differently, for example, running a job on a timer every few minutes, passing off to some other process, etc. Information technology should be changed depending on your own required implementation.

1. [HttpPost]
2. public  HttpResponseMessage UploadFile()
3. {
4. foreach  ( string  file in  Request.Files)
5.     {
6.         var FileDataContent = Request.Files[file];
7. if  (FileDataContent != null  && FileDataContent.ContentLength > 0)
8.         {
11.             var stream = FileDataContent.InputStream;
12.             var fileName = Path.GetFileName(FileDataContent.FileName);
13.             var UploadPath = Server.MapPath("~/App_Data/uploads" );
14.             Directory.CreateDirectory(UploadPath);
15. cord  path = Path.Combine(UploadPath, fileName);
16. try
17.             {
18. if  (System.IO.File.Exists(path))
19.                     System.IO.File.Delete(path);
20. using  (var fileStream = System.IO.File.Create(path))
21.                 {
22.                     stream.CopyTo(fileStream);
23.                 }
25.                 Shared.Utils UT =new  Shared.Utils();
26.                 UT.MergeFile(path);
27.             }
28. grab  (IOException ex)
29.             {
31.             }
32.         }
33.     }
34. return new  HttpResponseMessage()
35.     {
36.         StatusCode = System.Net.HttpStatusCode.OK,
37.         Content =new  StringContent( "File uploaded." )
38.     };
39. }

Each time we call the MergeFile method, information technology commencement checks to see if we take all of the file chunk parts required to merge the original file back together once again. It determines this by parsing the file-names. If all files are present, the method sorts them into the right guild, and then appends one to another until the original file that was separate, is dorsum together once more.

8. public bool  MergeFile( cord  FileName)
9. {
10. bool  rslt = faux ;
12. cord  partToken = ".part_" ;
13. string  baseFileName = FileName.Substring(0, FileName.IndexOf(partToken));
14. string  trailingTokens = FileName.Substring(FileName.IndexOf(partToken) + partToken.Length);
15. int  FileIndex = 0;
16. int  FileCount = 0;
17. int .TryParse(trailingTokens.Substring(0, trailingTokens.IndexOf( "." )), out  FileIndex);
18. int .TryParse(trailingTokens.Substring(trailingTokens.IndexOf( "." ) + 1), out  FileCount);
20. string  Searchpattern = Path.GetFileName(baseFileName) + partToken + "*" ;
21. string [] FilesList = Directory.GetFiles(Path.GetDirectoryName(FileName), Searchpattern);
25. if  (FilesList.Count() == FileCount)
26.     {
28. if  (!MergeFileManager.Case.InUse(baseFileName))
29.         {
30.             MergeFileManager.Instance.AddFile(baseFileName);
31. if  (File.Exists(baseFileName))
32.                 File.Delete(baseFileName);
35.             List<SortedFile> MergeList =new  Listing<SortedFile>();
36. foreach  ( string  File in  FilesList)
37.             {
38.                 SortedFile sFile =new  SortedFile();
39.                 sFile.FileName = File;
40.                 baseFileName = File.Substring(0, File.IndexOf(partToken));
41.                 trailingTokens = File.Substring(File.IndexOf(partToken) + partToken.Length);
42. int .TryParse(trailingTokens.
43.                    Substring(0, trailingTokens.IndexOf("." )), out  FileIndex);
44.                 sFile.FileOrder = FileIndex;
45.                 MergeList.Add together(sFile);
46.             }
48.             var MergeOrder = MergeList.OrderBy(due south => s.FileOrder).ToList();
49. using  (FileStream FS = new  FileStream(baseFileName, FileMode.Create))
50.             {
52. foreach  (var chunk in  MergeOrder)
53.                 {
54. try
55.                     {
56. using  (FileStream fileChunk =
57. new  FileStream(clamper.FileName, FileMode.Open))
58.                         {
59.                             fileChunk.CopyTo(FS);
60.                         }
61.                     }
62. catch  (IOException ex)
63.                     {
65.                     }
66.                 }
67.             }
68.             rslt =true ;
70.             MergeFileManager.Instance.RemoveFile(baseFileName);
71.         }
72.     }
73. return  rslt;
74. }

Using the file carve up on the customer-side, and file-merge on the server-side, we at present take a very workable solution for uploading large files in a more secure manner than merely sending up in one large block of data. For testing, I used some large image files converted to a BMP from a hubble picture hither.

That's information technology - Happy uploading !

duffeyseemusting.blogspot.com

Source: https://www.c-sharpcorner.com/article/upload-large-files-to-mvc-webapi-using-partitioning/

Share this post