SSL证书以及实现HTTP反向代理

注意： 本文内容于 2024-11-09 19:20:07 创建，可能不会在此平台上进行更新。如果您希望查看最新版本或更多相关内容，请访问原文地址：SSL证书以及实现HTTP反向代理。感谢您的关注与支持！

之前写的HTTP反向代理工具，在实际使用时，碰到反代失败的问题。跟踪了一下，才发现是由于对方使用了自签名SSL证书，导致发起HTTP请求时，验证失败。因此简单记录一下。

针对该问题的复现，从两个方面来展开

1. 理解SSL
2. 忽略SSL

一、理解SSL

1.1 HTTPS与SSL

SSL是用于加密传输的协议，也是最初的加密标准，目前已被TLS取代，但由于历史原因，大家还是会称为SSL。

HTTPS是HTTP上实现加密传输的协议，依赖SSL/TLS来确保安全性。

从不求甚解的角度来理解，HTTPS=HTTP+SSL/TLS

1.2 证书分类

SSL常见的证书分类有两种

• CA证书
• 自签名证书

这两者的区别如下

互联网服务使用的一般都是CA证书，由于操作系统已经内置了一系列根证书，当访问一个使用CA签发证书的HTTPS网站时，就不会出现“不安全连接”的警告。

而自签名证书，由于操作系统缺少对其的信任，访问就会被拦截了。此时服务提供方，需要给调用方提供自签名证书，以便调用方可以信任该连接。

1.3 OpenSSL生成自签名证书

1.3.1 扩展名说明

像我购买的CA证书，部署到Nginx时，一般都是.pem和.key文件。但在自己生成证书的过程中，发现还有.crt文件。直观的感受是，这些扩展名特别的混乱。经过查阅资料，下面简单记录这些扩展名的区别。

• crt: 存储证书(公钥)。该证书可提供给第三方使用，比如HTTPS客户端
• key: 私钥。该私钥文件只应给服务提供者使用。
• csr: 向证书颁发机构申请签署密钥的请求，不包含密钥本身。
• pem: 基于Base64编码的文本格式。它可以是上述任何文件。
• der: 基于二进制编码的文本格式。它可以是上述任何文件。

参考

ssl - Difference between pem, crt, key files - Stack Overflow

Difference between .pem and .crt and how to use them - Help - Let’s Encrypt Community Support

1.3.2 自签名证书

下面使用OpenSSL生成自签名的公钥和私钥证书。

# 生成一个2048位的RSA私钥，并保存到private.key文件中
openssl genrsa -out private.key 2048

# 根据私钥，生成一个证书签名请求
openssl req -new -key private.key -out request.csr

# X.509是SSL/TLS中最常用的公钥证书标准
# 通过私钥和证书签名请求，生成一个时效为365天的证书
openssl x509 -req -days 365 -in request.csr -signkey private.key -out public.pem

# 验证证书内容
openssl x509 -in public.pem -noout -text

二、忽略SSL

2.1 服务端部署证书

2.1.1 Nginx

以Nginx为例，部署证书

worker_processes 1;

events {
    worker_connections 1024;
}

http {
    include mime.types;
    default_type application/octet-stream;

    server {
        listen 443 ssl;
        server_name 10.0.0.1;

        ssl_certificate /usr/local/nginx/conf/cert/public.pem;
        ssl_certificate_key /usr/local/nginx/conf/cert/private.key;

        location / {
            root /usr/local/nginx/html;
            index index.html;
        }
    }
}

2.1.2 Vertx

使用Java中的Vertx 4.5.10版本开启HTTPServer

PemKeyCertOptions pemKeyCertOptions = new PemKeyCertOptions()//使用自签名证书开启ssl
        .addCertPath("/usr/local/nginx/conf/cert/public.pem")
        .addKeyPath("/usr/local/nginx/conf/cert/private.key");
Future<HttpServer> serverFuture = vertx.createHttpServer(new HttpServerOptions()
                .setSsl(true)
                .setKeyCertOptions(pemKeyCertOptions))
        //注册路由
        .requestHandler(router)
        .listen(port);
serverFuture.onComplete(re -> {
    if (re.succeeded()) {
        log.info("http server started on port {}", port);
    } else {
        log.error("http server failed to start", re.cause());
    }
});

2.2 客户端忽略校验

2.2.1 CURL

curl忽略ssl校验比较简单，添加-k参数即可。

curl -k "https://10.0.0.10:443"

2.2.2 Apache HttpPClient and OkHttpClient

设置忽略SSL的核心逻辑如下，具体的写法还需根据框架而定。

/**
 * 信任所有SSL证书，包括CA证书和自签名证书。实现效果类似于`curl -k`
 * @see <a href="https://blog.csdn.net/qq_20683411/article/details/142996223">Apache HttpClient 4.3.2 和 4.5.13 - 忽略证书问题_apache 4.3.5 忽略ssl-CSDN博客</a>
 */
public SSLContext trustAllCerts() {
    try {
        TrustManager[] trustManagers = {
                new X509TrustManager() {
                    @Override
                    public void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {

                    }

                    @Override
                    public void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {

                    }

                    @Override
                    public X509Certificate[] getAcceptedIssuers() {
                        return new X509Certificate[0];
                    }
                }
        };
        SSLContext sslContext = SSLContext.getInstance("SSL");
        sslContext.init(null, trustManagers, new SecureRandom());
        return sslContext;
    } catch (Exception ignore) {

    }
    return null;
}
    
public HostnameVerifier getNoopHostnameVerifier() {
    return new HostnameVerifier() {
        @Override
        public boolean verify(final String s, final SSLSession sslSession) {
            return true;
        }

        @Override
        public final String toString() {
            return "NO_OP";
        }
    };
}

三、HTTP反向代理

这个主要是带着学习的目的实现的。

• java8
• springboot2.5.14
• okhttp3

直接上源码meethigher/http-proxy-boot: 使用SpringBoot实现的开箱即用的HTTP反向代理工具

import okhttp3.*;
import okio.BufferedSink;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.net.ssl.*;
import javax.servlet.ServletException;
import javax.servlet.ServletInputStream;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.InputStream;
import java.security.SecureRandom;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.*;
import java.util.concurrent.TimeUnit;

/**
 * 该内容主要是想着学习一下底层HTTP反向代理的实现
 *
 * @author <a href="https://meethigher.top">chenchuancheng</a>
 * @see <a href="https://github.com/mitre/HTTP-Proxy-Servlet">mitre/HTTP-Proxy-Servlet: Smiley&#39;s HTTP Proxy implemented as a Java servlet</a>
 * @since 2024/11/09 22:43
 */
public class ProxyServlet extends HttpServlet {
    protected static final Logger log = LoggerFactory.getLogger(ProxyServlet.class);


    protected final OkHttpClient client;
    protected final String targetUrl; // 目标服务器信息
    protected final boolean corsControl; // 跨域控制。当为true时，跨域信息都由自身服务管理
    protected final boolean allowCORS; // 是否允许跨域。当corsControl为true时，该参数方可生效。
    protected final boolean logEnable; // 启用日志
    protected final boolean forwardIp; // 遵循代理规范，将实际调用方的ip和protocol传给目标服务器
    protected final boolean preserveHost; // 保留原host，这个仅对请求头有效。
    protected final boolean preserveCookie; // 保留原cookie。这个对请求头和响应头均有效。
    protected final String logFormat; // 日志格式

    /**
     * 跨域相关的响应头
     */
    protected final List<String> allowCORSHeaders = Arrays.asList(
            "access-control-allow-origin",//指定哪些域可以访问资源。可以是特定域名，也可以是通配符 *，表示允许所有域访问。
            "access-control-allow-methods",//指定允许的HTTP方法，如 GET、POST、PUT、DELETE 等。
            "access-control-allow-headers",//指定允许的请求头。
            "access-control-allow-credentials",//指定是否允许发送凭据（如Cookies）。值为 true 表示允许，且不能使用通配符 *。
            "access-control-expose-headers",//指定哪些响应头可以被浏览器访问。
            "access-control-max-age",//指定预检请求的结果可以被缓存的时间（以秒为单位）。
            "access-control-request-method",//在预检请求中使用，指示实际请求将使用的方法。
            "access-control-request-headers"//在预检请求中使用，指示实际请求将使用的自定义头。
    );

    /**
     * 不应该被复制的逐跳标头
     */
    protected final String[] hopByHopHeaders = new String[]{
            "Connection", "Keep-Alive", "Proxy-Authenticate", "Proxy-Authorization",
            "TE", "Trailers", "Transfer-Encoding", "Upgrade"};


    /**
     * 默认的日志格式
     */
    public static final String LOG_FORMAT_DEFAULT = "{method} -- {userAgent} -- {remoteAddr}:{remotePort} -- {source} --> {target} -- {statusCode} consumed {consumedMills} ms";


    protected void doLog(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse, long startMills) {
        if (logEnable) {
            String queryString = httpServletRequest.getQueryString();
            String logInfo = logFormat.replace("{method}", httpServletRequest.getMethod())
                    .replace("{userAgent}", httpServletRequest.getHeader("User-Agent"))
                    .replace("{remoteAddr}", httpServletRequest.getRemoteAddr())
                    .replace("{remotePort}", String.valueOf(httpServletRequest.getRemotePort()))
                    .replace("{source}", queryString == null ? httpServletRequest.getRequestURL() : httpServletRequest.getRequestURL() + "?" + queryString)
                    .replace("{target}", rewriteUrlFromRequest(httpServletRequest))
                    .replace("{statusCode}", String.valueOf(httpServletResponse.getStatus()))
                    .replace("{consumedMills}", String.valueOf(System.currentTimeMillis() - startMills));
            log.info("{}: {}", getServletName(), logInfo);
        }
    }

    /**
     * 是否包含逐跳标头
     */
    protected boolean containsHopByHopHeader(String name) {
        for (String header : hopByHopHeaders) {
            if (header.equalsIgnoreCase(name)) {
                return true;
            }
        }
        return false;
    }

    /**
     * 不考虑contextPath
     * 获取代理请求url，不包含queryparams
     */
    protected String getTargetUrl(HttpServletRequest request) {
        //request.getRequestURI();//包含contextPath的uri
        String uri = request.getPathInfo();//不包含contextPath的uri
        if (uri == null || uri.isEmpty()) {
            return targetUrl;
        } else {
            return targetUrl + uri;
        }
    }

    /**
     * 获取代理请求完整url，包含queryparams
     */
    protected String rewriteUrlFromRequest(HttpServletRequest request) {
        String targetUrl = getTargetUrl(request);
        String queryString = request.getQueryString();
        return queryString == null ? targetUrl : targetUrl + "?" + queryString;
    }

    /**
     * 将重定向的url，重写为代理服务器的地址
     */
    protected String rewriteUrlFromResponse(HttpServletRequest request, String locationUrl) {
        String targetUrl = getTargetUrl(request);
        if (locationUrl != null && locationUrl.startsWith(targetUrl)) {
            StringBuffer curUrl = request.getRequestURL();
            int pos;
            if ((pos = curUrl.indexOf("://")) >= 0) {
                if ((pos = curUrl.indexOf("/", pos + 3)) >= 0) {
                    curUrl.setLength(pos);
                }
            }
            curUrl.append(request.getContextPath());
            curUrl.append(request.getServletPath());
            curUrl.append(locationUrl, targetUrl.length(), locationUrl.length());
            return curUrl.toString();
        }
        return locationUrl;
    }

    public ProxyServlet(OkHttpClient client, String targetUrl, boolean corsControl, boolean allowCORS, boolean logEnable, String logFormat, boolean forwardIp, boolean preserveHost, boolean preserveCookie) {
        this.client = client;
        this.targetUrl = targetUrl;
        this.corsControl = corsControl;
        this.allowCORS = allowCORS;
        this.logEnable = logEnable;
        this.forwardIp = forwardIp;
        this.preserveHost = preserveHost;
        this.preserveCookie = preserveCookie;
        this.logFormat = logFormat;
    }

    public ProxyServlet(OkHttpClient client, String targetUrl) {
        this(client, targetUrl, false, false, true, LOG_FORMAT_DEFAULT, false, false, false);
    }

    /**
     * 根据代理的规定，通过请求头进行真实信息的传递
     * X-Forwarded-For: 传输实际调用者ip
     * X-Forwarded-Proto: 传输实际调用者请求协议
     */
    public void setXForwardedForHeader(HttpServletRequest request, Request.Builder requestBuilder) {
        if (forwardIp) {
            String forHeaderName = "X-Forwarded-For";
            String forHeader = request.getRemoteAddr();
            String existingForHeader = request.getHeader(forHeader);
            if (existingForHeader != null) {
                forHeader = existingForHeader + ", " + forHeader;
            }
            requestBuilder.header(forHeaderName, forHeader);
            String protoHeaderName = "X-Forwarded-Proto";
            String protoHeader = request.getScheme();
            requestBuilder.header(protoHeaderName, protoHeader);
        }
    }


    @Override
    protected void service(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) throws ServletException, IOException {
        long start = System.currentTimeMillis();
        try {
            if (httpServletRequest.getMethod().equalsIgnoreCase("options") && corsControl && allowCORS) {
                httpServletResponse.setHeader("Access-Control-Allow-Origin", httpServletRequest.getHeader("origin"));
                httpServletResponse.setHeader("Access-Control-Allow-Methods", "*");
                httpServletResponse.setHeader("Access-Control-Allow-Headers", "*");
                httpServletResponse.setHeader("Access-Control-Allow-Credentials", "true");
                httpServletResponse.setHeader("Access-Control-Expose-Headers", "*");
                httpServletResponse.setStatus(HttpServletResponse.SC_OK);
                return;
            }
            String method = httpServletRequest.getMethod();
            String proxyRequestUrl = rewriteUrlFromRequest(httpServletRequest);
            Request.Builder requestBuilder = getInitRequestBuilder(httpServletRequest, httpServletResponse);
            requestBuilder.url(proxyRequestUrl);
            if ("get".equalsIgnoreCase(method)) {
                requestBuilder.method(method, null);
            } else {
                // 根据HTTP规定，复制请求体
                RequestBody requestBody;
                if (httpServletRequest.getHeader("Content-Length") != null || httpServletRequest.getHeader("Transfer-Encoding") != null) {
                    try {
                        ServletInputStream inputStream = httpServletRequest.getInputStream();
                        requestBody = new StreamingRequestBody(MediaType.parse(httpServletRequest.getContentType()), inputStream);
                    } catch (Exception e) {
                        writeGatewayError(httpServletResponse, e.getMessage());
                        return;
                    }
                } else {
                    requestBody = RequestBody.create(null, new byte[0]);
                }
                requestBuilder.method(method, requestBody);
            }

            copyRequestHeaders(httpServletRequest, requestBuilder);
            setXForwardedForHeader(httpServletRequest, requestBuilder);
            try (Response response = client.newCall(requestBuilder.build()).execute()) {
                httpServletResponse.setStatus(response.code());
                copyResponseHeaders(httpServletRequest, httpServletResponse, response);
                if (response.code() == 304) {
                    // http状态码为304时，表示当客户端发起请求时，如果服务器发现请求的资源并没有自上次请求后发生任何更改，就会返回 304 状态码，同时不包含请求资源的实体内容。这意味着客户端可以继续使用缓存中的资源，从而避免不必要的数据传输，减少服务器负载和带宽消耗。
                    httpServletResponse.setIntHeader("Content-Length", 0);
                } else {
                    // 复制响应体
                    ResponseBody responseBody = response.body();
                    if (responseBody != null) {
                        ServletOutputStream os = httpServletResponse.getOutputStream();
                        InputStream is = responseBody.byteStream();
                        int len;
                        byte[] buffer = new byte[8192];
                        while ((len = is.read(buffer)) != -1) {
                            os.write(buffer, 0, len);
                        }
                        os.flush();
                    }
                }
            } catch (Exception e) {
                writeGatewayError(httpServletResponse, e.getMessage());
                return;
            }
        } finally {
            doLog(httpServletRequest, httpServletResponse, start);
        }
    }

    protected Request.Builder getInitRequestBuilder(HttpServletRequest request, HttpServletResponse response) {
        return new Request.Builder();
    }


    /**
     * 复制响应头
     */
    protected void copyResponseHeaders(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse, Response response) {
        Set<String> names = response.headers().names();
        // httpServletResponse不支持移除请求头，因此按需添加头
        Map<String, String> needSetHeaderMap = new LinkedHashMap<>();
        for (String name : names) {
            if (containsHopByHopHeader(name)) {
                continue;
            }
            if ("Location".equalsIgnoreCase(name)) {
                // 重写重定向头
                needSetHeaderMap.put(name, rewriteUrlFromResponse(httpServletRequest, response.header(name)));
            } else if ("Set-Cookie".equalsIgnoreCase(name) || "Set-Cookie2".equalsIgnoreCase(name)) {
                // 保存Cookie信息
                if (preserveCookie) {
                    needSetHeaderMap.put(name, response.header(name));
                }
            } else {
                needSetHeaderMap.put(name, response.header(name));
            }
        }

        /**
         * 跨域控制
         */
        if (corsControl) {
            /**
             * 1. 清空所有与跨域相关的响应头
             * 2. 如果允许跨域，则添加跨域允许响应头
             */
            Iterator<Map.Entry<String, String>> iterator = needSetHeaderMap.entrySet().iterator();
            while (iterator.hasNext()) {
                Map.Entry<String, String> entry = iterator.next();
                String keyIgnoreCase = entry.getKey().toLowerCase(Locale.ROOT);
                if (allowCORSHeaders.contains(keyIgnoreCase)) {
                    iterator.remove();
                }
            }
            if (allowCORS) {
                needSetHeaderMap.put("Access-Control-Allow-Origin", httpServletRequest.getHeader("origin"));
                needSetHeaderMap.put("Access-Control-Allow-Methods", "*");
                needSetHeaderMap.put("Access-Control-Allow-Headers", "*");
                needSetHeaderMap.put("Access-Control-Allow-Credentials", "true");
                needSetHeaderMap.put("Access-Control-Expose-Headers", "*");
            }
        }
        for (String header : needSetHeaderMap.keySet()) {
            httpServletResponse.setHeader(header, needSetHeaderMap.get(header));
        }
    }

    /**
     * 复制请求头
     */
    protected void copyRequestHeaders(HttpServletRequest httpServletRequest, Request.Builder requestBuilder) {
        Enumeration<String> headerNames = httpServletRequest.getHeaderNames();
        while (headerNames.hasMoreElements()) {
            String key = headerNames.nextElement();
            if ("host".equalsIgnoreCase(key)) {
                if (preserveHost) {
                    requestBuilder.header(key, httpServletRequest.getHeader(key));
                }
            } else if ("cookie".equalsIgnoreCase(key)) {
                if (preserveCookie) {
                    requestBuilder.header(key, httpServletRequest.getHeader(key));
                }
            } else {
                requestBuilder.header(key, httpServletRequest.getHeader(key));
            }
        }
    }


    protected void writeGatewayError(HttpServletResponse httpServletResponse, String msg) {
        httpServletResponse.setStatus(502);
        httpServletResponse.setContentType("text/html;charset=utf-8");
        try {
            httpServletResponse.getWriter().write(msg);
        } catch (Exception ignore) {

        }
    }


    /**
     * okhttpclient的流式请求体
     * 节省应用内存，通过流式传输数据，只会在网络传输过程中按需读取数据，而不会将整个请求体加载到内存。
     */
    class StreamingRequestBody extends RequestBody {

        private final MediaType contentType;
        private final InputStream inputStream;

        public StreamingRequestBody(MediaType contentType, InputStream inputStream) {
            this.contentType = contentType;
            this.inputStream = inputStream;
        }

        @Override
        public MediaType contentType() {
            return contentType;
        }

        @Override
        public void writeTo(BufferedSink sink) throws IOException {
            // 每次读8KB的数据
            byte[] buffer = new byte[8192];
            int bytesRead;
            while ((bytesRead = inputStream.read(buffer)) != -1) {
                sink.write(buffer, 0, bytesRead);
            }
        }
    }


    private static OkHttpClient okHttpClient;

    public synchronized static OkHttpClient okHttpClient() {
        if (okHttpClient == null) {
            OkHttpClient.Builder builder = new OkHttpClient().newBuilder()
                    .connectionPool(new ConnectionPool(2, 60, TimeUnit.SECONDS))
                    .connectTimeout(10, TimeUnit.SECONDS)
                    .readTimeout(30, TimeUnit.SECONDS)
                    .writeTimeout(30, TimeUnit.SECONDS);
            builder.followRedirects(false);
            builder.followSslRedirects(true);
            builder.hostnameVerifier(new HostnameVerifier() {
                @Override
                public boolean verify(final String s, final SSLSession sslSession) {
                    return true;
                }

                @Override
                public final String toString() {
                    return "NO_OP";
                }
            });
            try {
                X509TrustManager x509TrustManager = new X509TrustManager() {
                    @Override
                    public void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {

                    }

                    @Override
                    public void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {

                    }

                    @Override
                    public X509Certificate[] getAcceptedIssuers() {
                        return new X509Certificate[0];
                    }
                };
                TrustManager[] trustManagers = {
                        x509TrustManager
                };
                SSLContext sslContext = SSLContext.getInstance("SSL");
                sslContext.init(null, trustManagers, new SecureRandom());
                builder.sslSocketFactory(sslContext.getSocketFactory(), x509TrustManager);
            } catch (Exception ignore) {

            }
            okHttpClient = builder.build();
        }
        return okHttpClient;
    }
}

​

​