Understanding multipart/form-data in HTTP protocol

I encountered the problem of encapsulating the Media type multipart/form-data when writing a generic HTTP component. This article briefly introduces the definition, application and simple implementation of the media type multipart/form-data in the HTTP protocol.

Definition of multipart/form-data

The media type multipart/form-data follows the multipart MIME data stream definition (which can be found in Section 5.1 - RFC2046), which roughly means that the data body of the media type multipart/form-data consists of multiple parts separated by a fixed Boundary.

multipart/form-data request body layout

The layout of the multipart/form-data request body is as follows.

# 请求头 - 这个是必须的，需要指定Content-Type为multipart/form-data，指定唯一边界值
Content-Type: multipart/form-data; boundary=${Boundary}

# 请求体
--${Boundary}
Content-Disposition: form-data; name="name of file"
Content-Type: application/octet-stream

bytes of file
--${Boundary}
Content-Disposition: form-data; name="name of pdf"; filename="pdf-file.pdf"
Content-Type: application/octet-stream

bytes of pdf file
--${Boundary}
Content-Disposition: form-data; name="key"
Content-Type: text/plain;charset=UTF-8

text encoded in UTF-8
--${Boundary}--

The most obvious differences between the media type multipart/form-data versus other media types such as application/x-www-form-urlencoded are

The Content-Type attribute of the request header, in addition to specifying multipart/form-data, also requires the definition of the boundary parameter
The request line data in the request body is composed of multiple parts, and the value pattern of the boundary parameter - ${Boundary} is used to separate each individual division
The request header Content-Disposition: form-data; name="${PART_NAME}"; must be present in each section, where ${PART_NAME} needs to be URL encoded, and the filename field can be used to indicate the name of the file, but it is less binding than the name attribute (as there is no confirmation that the local file is available or objectionable)
Each part can define Content-Type and the data body of that part separately
The request body ends with the value pattern -${Boundary}-- of the boundary parameter

RFC7578 mentions two multipart/form-data expired use, one is the use of Content-Transfer-Encoding request header, here also do not expand its use, the second is the request body of a single form attribute transmission of multiple binary file way recommended to switch to multipart/mixed (a “name” corresponds to multiple binary file scenario)

Special.

If the content of a part is text, its Content-Type is text/plain, you can specify the corresponding character set, such as Content-Type: text/plain;charset=UTF-8
The default charset can be specified via the charset attribute, which is used as follows.

Content-Disposition: form-data; name="_charset_"

UTF-8
--ABCDE--
Content-Disposition: form-data; name="field"

...text encoded in UTF-8...
ABCDE--

Boundary parameter value statute

The Boundary parameter takes the following value statute.

The value of the Boundary must begin with a double horizontal bar – in the middle of the English language, this – is called the leading hyphen
The value of the Boundary must not contain more than 70 characters in addition to the leading hyphen.
The value of the Boundary must not contain characters that are disabled by the HTTP protocol or the URL, such as the colon: etc.
Each – ${Boundary} before the default mandatory must be CRLF, if a part of the text type request body ends with CRLF, then in the request body of the secondary system format, there must be two CRLF explicitly, if a part of the request body does not end with CRLF, can only exist a CRLF, these two cases are called the separator of the explicit type and implicit type, said more abstract, see the following example.

# 请求头
Content-type: multipart/data; boundary="--abcdefg"

--abcdefg
Content-Disposition: form-data; name="x"
Content-type: text/plain; charset=ascii

It does NOT end with a linebreak # <=== 这里没有CRLF，隐式类型
--abcdefg
Content-Disposition: form-data; name="y"
Content-type: text/plain; charset=ascii

It DOES end with a linebreak # <=== 这里有CRLF，显式类型

--abcdefg

## 直观看隐式类型的CRLF
It does NOT end with a linebreak CRLF --abcdefg

## 直观看显式类型的CRLF
It DOES end with a linebreak CRLF CRLF --abcdefg

Implementing POST requests for multipart/form-data media types

Here only for the low JDK version of HttpURLConnection and high JDK version of the built-in HttpClient to write multipart/form-data media type of POST requests HTTP client, others such as custom Socket implementation can be completed along similar lines. First introduce org.springframework.boot:spring-boot-starter-web:2.6.0 to do a simple controller method.

@RestController
public class TestController {

    @PostMapping(path = "/test")
    public ResponseEntity<?> test(MultipartHttpServletRequest request) {
        return ResponseEntity.ok("ok");
    }
}

Postman’s simulated request is as follows.

The request parameters obtained by the backend controller are as follows.

The client written later can call this interface directly for debugging.

Module that encapsulates the conversion of request bodies into byte containers

Here the boundary values are all implemented explicitly, and the boundary values are generated directly with a fixed prefix plus the UUID. Some simplifications are made in the simple implementation.

Only text form data and binary (file) form data are considered for submission
Based on the previous point, each section explicitly specifies Content-Type as the request header
Text encoding is fixed to UTF-8

Write a MultipartWriter.

public class MultipartWriter {

    private static final Charset DEFAULT_CHARSET = StandardCharsets.UTF_8;
    private static final byte[] FIELD_SEP = ": ".getBytes(StandardCharsets.ISO_8859_1);
    private static final byte[] CR_LF = "\r\n".getBytes(StandardCharsets.ISO_8859_1);
    private static final String TWO_HYPHENS_TEXT = "--";
    private static final byte[] TWO_HYPHENS = TWO_HYPHENS_TEXT.getBytes(StandardCharsets.ISO_8859_1);
    private static final String CONTENT_DISPOSITION_KEY = "Content-Disposition";
    private static final String CONTENT_TYPE_KEY = "Content-Type";
    private static final String DEFAULT_CONTENT_TYPE = "multipart/form-data; boundary=";
    private static final String DEFAULT_BINARY_CONTENT_TYPE = "application/octet-stream";
    private static final String DEFAULT_TEXT_CONTENT_TYPE = "text/plain;charset=UTF-8";
    private static final String DEFAULT_CONTENT_DISPOSITION_VALUE = "form-data; name=\"%s\"";
    private static final String FILE_CONTENT_DISPOSITION_VALUE = "form-data; name=\"%s\"; filename=\"%s\"";

    private final Map<String, String> headers = new HashMap<>(8);
    private final List<AbstractMultipartPart> parts = new ArrayList<>();
    private final String boundary;

    private MultipartWriter(String boundary) {
        this.boundary = Objects.isNull(boundary) ? TWO_HYPHENS_TEXT +
                UUID.randomUUID().toString().replace("-", "") : boundary;
        this.headers.put(CONTENT_TYPE_KEY, DEFAULT_CONTENT_TYPE + this.boundary);
    }

    public static MultipartWriter newMultipartWriter(String boundary) {
        return new MultipartWriter(boundary);
    }

    public static MultipartWriter newMultipartWriter() {
        return new MultipartWriter(null);
    }

    public MultipartWriter addHeader(String key, String value) {
        if (!CONTENT_TYPE_KEY.equalsIgnoreCase(key)) {
            headers.put(key, value);
        }
        return this;
    }

    public MultipartWriter addTextPart(String name, String text) {
        parts.add(new TextPart(String.format(DEFAULT_CONTENT_DISPOSITION_VALUE, name), DEFAULT_TEXT_CONTENT_TYPE, this.boundary, text));
        return this;
    }

    public MultipartWriter addBinaryPart(String name, byte[] bytes) {
        parts.add(new BinaryPart(String.format(DEFAULT_CONTENT_DISPOSITION_VALUE, name), DEFAULT_BINARY_CONTENT_TYPE, this.boundary, bytes));
        return this;
    }

    public MultipartWriter addFilePart(String name, File file) {
        parts.add(new FilePart(String.format(FILE_CONTENT_DISPOSITION_VALUE, name, file.getName()), DEFAULT_BINARY_CONTENT_TYPE, this.boundary, file));
        return this;
    }

    private static void writeHeader(String key, String value, OutputStream out) throws IOException {
        writeBytes(key, out);
        writeBytes(FIELD_SEP, out);
        writeBytes(value, out);
        writeBytes(CR_LF, out);
    }

    private static void writeBytes(String text, OutputStream out) throws IOException {
        out.write(text.getBytes(DEFAULT_CHARSET));
    }

    private static void writeBytes(byte[] bytes, OutputStream out) throws IOException {
        out.write(bytes);
    }

    interface MultipartPart {

        void writeBody(OutputStream os) throws IOException;
    }

    @RequiredArgsConstructor
    public static abstract class AbstractMultipartPart implements MultipartPart {

        protected final String contentDispositionValue;
        protected final String contentTypeValue;
        protected final String boundary;

        protected String getContentDispositionValue() {
            return contentDispositionValue;
        }

        protected String getContentTypeValue() {
            return contentTypeValue;
        }

        protected String getBoundary() {
            return boundary;
        }

        public final void write(OutputStream out) throws IOException {
            writeBytes(TWO_HYPHENS, out);
            writeBytes(getBoundary(), out);
            writeBytes(CR_LF, out);
            writeHeader(CONTENT_DISPOSITION_KEY, getContentDispositionValue(), out);
            writeHeader(CONTENT_TYPE_KEY, getContentTypeValue(), out);
            writeBytes(CR_LF, out);
            writeBody(out);
            writeBytes(CR_LF, out);
        }
    }

    public static class TextPart extends AbstractMultipartPart {

        private final String text;

        public TextPart(String contentDispositionValue,
                        String contentTypeValue,
                        String boundary,
                        String text) {
            super(contentDispositionValue, contentTypeValue, boundary);
            this.text = text;
        }

        @Override
        public void writeBody(OutputStream os) throws IOException {
            os.write(text.getBytes(DEFAULT_CHARSET));
        }

        @Override
        protected String getContentDispositionValue() {
            return contentDispositionValue;
        }

        @Override
        protected String getContentTypeValue() {
            return contentTypeValue;
        }
    }

    public static class BinaryPart extends AbstractMultipartPart {

        private final byte[] content;

        public BinaryPart(String contentDispositionValue,
                          String contentTypeValue,
                          String boundary,
                          byte[] content) {
            super(contentDispositionValue, contentTypeValue, boundary);
            this.content = content;
        }

        @Override
        public void writeBody(OutputStream out) throws IOException {
            out.write(content);
        }
    }

    public static class FilePart extends AbstractMultipartPart {

        private final File file;

        public FilePart(String contentDispositionValue,
                        String contentTypeValue,
                        String boundary,
                        File file) {
            super(contentDispositionValue, contentTypeValue, boundary);
            this.file = file;
        }

        @Override
        public void writeBody(OutputStream out) throws IOException {
            try (InputStream in = new FileInputStream(file)) {
                final byte[] buffer = new byte[4096];
                int l;
                while ((l = in.read(buffer)) != -1) {
                    out.write(buffer, 0, l);
                }
                out.flush();
            }
        }
    }

    public void forEachHeader(BiConsumer<String, String> consumer) {
        headers.forEach(consumer);
    }

    public void write(OutputStream out) throws IOException {
        if (!parts.isEmpty()) {
            for (AbstractMultipartPart part : parts) {
                part.write(out);
            }
        }
        writeBytes(TWO_HYPHENS, out);
        writeBytes(this.boundary, out);
        writeBytes(TWO_HYPHENS, out);
        writeBytes(CR_LF, out);
    }
}

This class has encapsulated three different types of partial request body implementations. The forEachHeader() method is used to iterate through the request headers, and the final write() method is used to write the request body to the OutputStream.

HttpURLConnection implementation

The implementation code is as follows (minimal implementation only, without consideration of fault tolerance and exception handling).

public class HttpURLConnectionApp {

    private static final String URL = "http://localhost:9099/test";

    public static void main(String[] args) throws Exception {
        MultipartWriter writer = MultipartWriter.newMultipartWriter();
        writer.addTextPart("name", "throwable")
                .addTextPart("domain", "vlts.cn")
                .addFilePart("ico", new File("I:\\doge_favicon.ico"));
        DataOutputStream requestPrinter = new DataOutputStream(System.out);
        writer.write(requestPrinter);
        HttpURLConnection connection = (HttpURLConnection) new java.net.URL(URL).openConnection();
        connection.setRequestMethod("POST");
        connection.addRequestProperty("Connection", "Keep-Alive");
        // 设置请求头
        writer.forEachHeader(connection::addRequestProperty);
        connection.setDoInput(true);
        connection.setDoOutput(true);
        connection.setConnectTimeout(10000);
        connection.setReadTimeout(10000);
        DataOutputStream out = new DataOutputStream(connection.getOutputStream());
        // 设置请求体
        writer.write(out);
        StringBuilder builder = new StringBuilder();
        BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream(), StandardCharsets.UTF_8));
        String line;
        while (Objects.nonNull(line = reader.readLine())) {
            builder.append(line);
        }
        int responseCode = connection.getResponseCode();
        reader.close();
        out.close();
        connection.disconnect();
        System.out.printf("响应码:%d,响应内容:%s\n", responseCode, builder);
    }
}

Implementation response results.

`1`	`响应码:200,响应内容:ok`

You can try adding two lines of code to print the request body.

MultipartWriter writer = MultipartWriter.newMultipartWriter();
writer.addTextPart("name", "throwable")
        .addTextPart("domain", "vlts.cn")
        .addFilePart("ico", new File("I:\\doge_favicon.ico"));
DataOutputStream requestPrinter = new DataOutputStream(System.out);
writer.write(requestPrinter);

Console output as follows.

JDK built-in HttpClient implementation

JDK11+ built-in HTTP client implementation, the specific entrance is java.net.http.HttpClient, the implementation code is as follows.

public class HttpClientApp {

    private static final String URL = "http://localhost:9099/test";

    public static void main(String[] args) throws Exception {
        HttpClient httpClient = HttpClient.newBuilder()
                .connectTimeout(Duration.of(10, ChronoUnit.SECONDS))
                .build();
        MultipartWriter writer = MultipartWriter.newMultipartWriter();
        writer.addTextPart("name", "throwable")
                .addTextPart("domain", "vlts.cn")
                .addFilePart("ico", new File("I:\\doge_favicon.ico"));
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        writer.write(out);
        HttpRequest.Builder requestBuilder = HttpRequest.newBuilder();
        writer.forEachHeader(requestBuilder::header);
        HttpRequest request = requestBuilder.uri(URI.create(URL))
                .method("POST", HttpRequest.BodyPublishers.ofByteArray(out.toByteArray()))
                .build();
        HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
        System.out.printf("响应码:%d,响应内容:%s\n", response.statusCode(), response.body());
    }
}

The built-in HTTP components are almost all using Reactive programming model, using a relatively low-level API, which is more flexible but not as easy to use.

Summary

The media type multipart/form-data is commonly used in HTTP requests under the POST method, and is relatively uncommon as an HTTP response.

Table of Contents