我要评论什么是大文件上传
大文件上传通常指上传超过几百mb甚至几个gb的文件。与普通文件上传相比，大文件上传面临以下挑战：
- 内存限制 - 一次性加载整个文件到内存会导致内存溢出
- 网络稳定性 - 上传过程中网络中断需要能够断点续传
- 超时问题 - 长时间上传可能导致连接超时
- 进度监控 - 需要实时显示上传进度
- 文件校验 - 确保文件完整性和安全性
解决方案：分片上传
大文件上传的核心思想是将文件分割成多个小块，分别上传，最后在服务器端合并。
前端代码示例 (html + javascript)
<!doctype html>
<html>
<head>
<title>大文件上传</title>
</head>
<body>
<input type="file" id="fileinput" />
<button onclick="uploadfile()">开始上传</button>
<div id="progress"></div>
<script>
const chunk_size = 2 * 1024 * 1024; // 2mb
async function uploadfile() {
const fileinput = document.getelementbyid('fileinput');
const file = fileinput.files[0];
if (!file) {
alert('请选择文件');
return;
}
const totalchunks = math.ceil(file.size / chunk_size);
const filemd5 = await calculatefilemd5(file);
// 检查文件是否已上传过
const checkresult = await checkfileexists(file.name, filemd5, file.size);
if (checkresult.uploaded) {
alert('文件已存在');
return;
}
let uploadedchunks = checkresult.uploadedchunks || [];
for (let chunkindex = 0; chunkindex < totalchunks; chunkindex++) {
// 跳过已上传的分片
if (uploadedchunks.includes(chunkindex)) {
updateprogress(chunkindex + 1, totalchunks);
continue;
}
const chunk = file.slice(chunkindex * chunk_size, (chunkindex + 1) * chunk_size);
const formdata = new formdata();
formdata.append('file', chunk);
formdata.append('chunkindex', chunkindex);
formdata.append('totalchunks', totalchunks);
formdata.append('filename', file.name);
formdata.append('filemd5', filemd5);
try {
await uploadchunk(formdata);
updateprogress(chunkindex + 1, totalchunks);
} catch (error) {
console.error(`分片 ${chunkindex} 上传失败:`, error);
alert('上传失败');
return;
}
}
// 所有分片上传完成，请求合并
await mergechunks(file.name, filemd5, totalchunks);
alert('上传完成');
}
function uploadchunk(formdata) {
return fetch('/upload/chunk', {
method: 'post',
body: formdata
}).then(response => {
if (!response.ok) {
throw new error('上传失败');
}
return response.json();
});
}
function checkfileexists(filename, filemd5, filesize) {
return fetch(`/upload/check?filename=${filename}&filemd5=${filemd5}&filesize=${filesize}`)
.then(response => response.json());
}
function mergechunks(filename, filemd5, totalchunks) {
return fetch('/upload/merge', {
method: 'post',
headers: {
'content-type': 'application/json',
},
body: json.stringify({
filename: filename,
filemd5: filemd5,
totalchunks: totalchunks
})
}).then(response => response.json());
}
function updateprogress(current, total) {
const progress = document.getelementbyid('progress');
const percentage = math.round((current / total) * 100);
progress.innerhtml = `上传进度: ${percentage}%`;
}
// 计算文件md5（简化版，实际应使用更可靠的库）
async function calculatefilemd5(file) {
// 这里使用简单的文件名+大小模拟md5
// 实际项目中应使用 spark-md5 等库
return btoa(file.name + file.size).replace(/[^a-za-z0-9]/g, '');
}
</script>
</body>
</html>
后端java代码示例 (spring boot)
配置文件上传设置
@configuration
public class uploadconfig {
@bean
public multipartconfigelement multipartconfigelement() {
multipartconfigfactory factory = new multipartconfigfactory();
factory.setmaxfilesize("10gb");
factory.setmaxrequestsize("10gb");
return factory.createmultipartconfig();
}
}
文件上传控制器
@restcontroller
@requestmapping("/upload")
public class fileuploadcontroller {
@value("${file.upload-dir:/tmp/uploads}")
private string uploaddir;
/**
* 检查文件是否存在
*/
@getmapping("/check")
public responseentity<checkresult> checkfile(
@requestparam string filename,
@requestparam string filemd5,
@requestparam long filesize) {
string filepath = paths.get(uploaddir, filemd5, filename).tostring();
file file = new file(filepath);
checkresult result = new checkresult();
// 如果文件已存在
if (file.exists() && file.length() == filesize) {
result.setuploaded(true);
return responseentity.ok(result);
}
// 检查已上传的分片
string chunkdir = getchunkdir(filemd5);
file chunkfolder = new file(chunkdir);
if (!chunkfolder.exists()) {
result.setuploaded(false);
result.setuploadedchunks(new arraylist<>());
return responseentity.ok(result);
}
list<integer> uploadedchunks = arrays.stream(chunkfolder.listfiles())
.map(f -> integer.parseint(f.getname()))
.collect(collectors.tolist());
result.setuploaded(false);
result.setuploadedchunks(uploadedchunks);
return responseentity.ok(result);
}
/**
* 上传文件分片
*/
@postmapping("/chunk")
public responseentity<uploadresult> uploadchunk(
@requestparam("file") multipartfile file,
@requestparam integer chunkindex,
@requestparam integer totalchunks,
@requestparam string filename,
@requestparam string filemd5) {
try {
// 创建分片目录
string chunkdir = getchunkdir(filemd5);
file chunkfolder = new file(chunkdir);
if (!chunkfolder.exists()) {
chunkfolder.mkdirs();
}
// 保存分片文件
file chunkfile = new file(chunkdir + file.separator + chunkindex);
file.transferto(chunkfile);
uploadresult result = new uploadresult();
result.setsuccess(true);
result.setmessage("分片上传成功");
return responseentity.ok(result);
} catch (exception e) {
uploadresult result = new uploadresult();
result.setsuccess(false);
result.setmessage("分片上传失败: " + e.getmessage());
return responseentity.status(httpstatus.internal_server_error).body(result);
}
}
/**
* 合并文件分片
*/
@postmapping("/merge")
public responseentity<mergeresult> mergechunks(@requestbody mergerequest request) {
try {
string chunkdir = getchunkdir(request.getfilemd5());
string filename = request.getfilename();
string filepath = paths.get(uploaddir, request.getfilemd5(), filename).tostring();
// 创建目标文件
file targetfile = new file(filepath);
file parentdir = targetfile.getparentfile();
if (!parentdir.exists()) {
parentdir.mkdirs();
}
// 合并分片
try (fileoutputstream fos = new fileoutputstream(targetfile)) {
for (int i = 0; i < request.gettotalchunks(); i++) {
file chunkfile = new file(chunkdir + file.separator + i);
try (fileinputstream fis = new fileinputstream(chunkfile)) {
byte[] buffer = new byte[1024];
int len;
while ((len = fis.read(buffer)) > 0) {
fos.write(buffer, 0, len);
}
}
// 删除分片文件
chunkfile.delete();
}
}
// 删除分片目录
new file(chunkdir).delete();
mergeresult result = new mergeresult();
result.setsuccess(true);
result.setmessage("文件合并成功");
result.setfilepath(filepath);
return responseentity.ok(result);
} catch (exception e) {
mergeresult result = new mergeresult();
result.setsuccess(false);
result.setmessage("文件合并失败: " + e.getmessage());
return responseentity.status(httpstatus.internal_server_error).body(result);
}
}
private string getchunkdir(string filemd5) {
return paths.get(uploaddir, "chunks", filemd5).tostring();
}
}
数据传输对象
@data
public class checkresult {
private boolean uploaded;
private list<integer> uploadedchunks;
}
@data
public class uploadresult {
private boolean success;
private string message;
}
@data
public class mergerequest {
private string filename;
private string filemd5;
private integer totalchunks;
}
@data
public class mergeresult {
private boolean success;
private string message;
private string filepath;
}
应用配置
# application.properties spring.servlet.multipart.max-file-size=10gb spring.servlet.multipart.max-request-size=10gb file.upload-dir=/data/uploads
关键技术点
- 分片上传：将大文件分割成小块，分别上传
- 断点续传：记录已上传的分片，网络中断后可以从中断处继续
- 文件校验：通过md5验证文件完整性
- 进度监控：实时显示上传进度
- 内存优化：流式处理，避免内存溢出
优化建议
- 增加重试机制：网络异常时自动重试
- 并行上传：同时上传多个分片提高速度
- 压缩传输：对分片进行压缩减少网络传输量
- 安全验证：添加身份验证和文件类型检查
- 分布式存储：支持分布式文件系统存储
这种方案可以有效解决大文件上传的各种问题，提供稳定可靠的上传体验。
到此这篇关于java大文件上传(分片上传+断点续传)的解决方案详解的文章就介绍到这了,更多相关java大文件上传内容请搜索代码网以前的文章或继续浏览下面的相关文章希望大家以后多多支持代码网！
发表评论