I am going to give a talk highlighting elementary concepts in online advertising. It is a good time to review all the basics. DSP Demand-side platforms receive an ad request that they must answer by a time set by the SSP or ad exchange. The time allowed can be as low as 100 ms and range up to a few seconds. DSPs decide whether they want to bid. If they do, they must select an ad, determine a bid price, and return their offer to the ad exchange.

In advertising and other online marketing fields, a common practice is to distribute URLs to the clients so that they can report user action through a simple GET request.

Those parameters are fake. But you got the idea. Longer URLs mean higher bandwidth consumption. Moreover, if your URL is too long, you will risk triggering Error 413: request entity too large. I know nginx throws it.

We have carried out the first or maybe last round of rewrite of the Juhui DSP/SSP/ADX with Typescript. A few goals and non-goals have been set for this labor-intensive process. Goals Strong typed. Strong API contracts. Holistic Observability with logging, tracing, and metrics. Layered Architecture and dedicated component. Testcases can be run by anyone. Stateless over stateful. Improved readability. Proper code deprecation process. Non-Goals Use fancy technology.

DMP stands for Data Management Platform, and per Wikipedia, it is a technology platform used for collecting and managing data, mainly for digital marketing purposes. To put it simply, DMP is a collection of data about user attributes. As you might expect a DMP may contain more than millions of data. The DMP I worked on contained nearly a billion. Open source toolings such as Elasticsearch provided a good average baseline for data analytics, and reasonable response time for general-purpose querying.

I have done some quick’n’dirty bid data processing with Apache Beam in past weeks. As someone who try to stay away from JVM, I am not a big data expert at all. However the working with apache beam is a blast (Using JAVA!). The lazy execution style feels right a home. I love wiring a various component into a topology, and only start processing when data arrives. I once heard a Haskell fan said that she thought lazy execution was what set Haskell apart from other programing languages.

The Clean Architecture is more like a fantasy. It is like a wonderland we will never be in. I would enjoy and appreciate a work where all business domains, application domains, and transports and UIs, etc. are all orthogonal. In reality, they are all wired together like a mess. It is no mess created by engineer though. The world is chaotic by itself, so do all the business around it. Imagine your boss tell your team to add a new button to your website, and you find your business rule doesn’t cover this button what so ever, so you have to update your JSON API, add new test cases, and add new event to your event sourcing bus, and change whatever is affected all the way down to the database level.

DMP stands for Data Management Platform, and per Wikipedia, it is a technology platform used for collecting and managing data, mainly for digital marketing purposes. To put it simply, DMP is a collection of data about user attributes. As you might expect a DMP may contain more than millions of data. The DMP I worked on contained nearly a billion. Open source toolings such as Elasticsearch provided a good average baseline for data analytics, and reasonable response time for general-purpose querying.

I have been on both sides. Sometimes people are amazed at our project grounding up from zero so soon, and some other times people are questioning why our project progress is that slow. It was said that compiled/strong-typed languages were faster at runtime, but slower to code. While it is true to a certain degree, it rarely matters in commercial applications. There was also a common belief that enforcing code qualities by means of TDD/Code review/testing-in-general would increase the time to launch.

注解是元编程的一种。元编程从字面意思上说就是编写程序的程序。和普通编程一样，注解在给我们带来便捷的同时，如果使用不当，也有可能造成可读性、可维护性下降等问题。 在某些注解中，可能有很多配置项，比如： //这还不是一个特别夸张的例子 @CircuitBreaker(timeout=0.05, failCounter=1, successCounter=1, fallback="App\Service\UserService::searchFallback") 如果我们的代码里用很多这样复杂的注解，就会引发以下几个问题： 注解中可使用的数据类型表达能力有限，比如必须用方法的字符串全名来表达方法，容易出错。 离开了IDE的帮助，长注解的可读性变得很差。（比如在GitHub上） 同样配置的注解多个地方使用，修改时要改很多地方。 这里我向大家推荐通过继承的方式配置Hyperf内的注解。 下面是一个继承CircuitBreaker（熔断器）注解的例子。 <?php ... /** * @Annotation * @Target({"METHOD"}) * * Shorthand for CircuitBreaker(timeout=0.05, failCounter=1, successCounter=1, fallback="App\Service\UserService::searchFallback") */ class FooCircuitBreakerAnnotation extends CircuitBreakerAnnotation { /** * @var float */ public $timeout = 0.05; /** * @var string */ public $fallback = UserService::class.'::searchFallback'; /** * The counter required to reset to a close state. * @var int */ public $successCounter = 1; /** * The counter required to reset to a open state.

The Clean Architecture is more like a fantasy. It is like a wonderland we will never be in. I would enjoy and appreciate a work where all business domains, application domains, and transports and UIs, etc. are all orthogonal. In reality, they are all wired together like a mess. It is no mess created by engineer though. The world is chaotic by itself, so do all the business around it. Imagine your boss tell your team to add a new button to your website, and you find your business rule doesn’t cover this button what so ever, so you have to update your JSON API, add new test cases, and add new event to your event sourcing bus, and change whatever is affected all the way down to the database level.

As the job demand is going nowhere in a decade, a lot of new guys want to get into programming. There are numerous courses online, teaching a variety of skills in programming. There are also new shinning frameworks climbing up GitHub trending every day. I see a lot of people, typically newly graduated students, get confused. There seems to be an endless stream of knowledge to learn. How many skills are enough?

We have carried out the first or maybe last round of rewrite of the Juhui DSP/SSP/ADX with Typescript. A few goals and non-goals have been set for this labor-intensive process. Goals Strong typed. Strong API contracts. Holistic Observability with logging, tracing, and metrics. Layered Architecture and dedicated component. Testcases can be run by anyone. Stateless over stateful. Improved readability. Proper code deprecation process. Non-Goals Use fancy technology.

Over the days I have been thinking about improving my coding one step further. As I was refactoring some company work, this idea came to me. One of the least mentioned traits of good code is that you can easily remove a slice of them (from a larger project). Think about that. Removability means components are modular, architecture is layered, and abstractions are aptly interfaced. It also means Single Responsibility is enforced, so removing one thing won’t jeopardize another.

Came across this pic from twitter. If I have an adequate number of audience, I would like to make poll out of it. What do you think about this piece of code? A: It is good code. B: It is bad code. I have never done react before (Or I have, but not professional). However this piece of code is very readable to me. If I was tasked with the job to maintain this code, I would appreciate the original author for coding in this fashion.

Sometimes in your daily programming life, you would want to inject different object instances based on the current route or module. For example, you want to connect to Database Foo for route /foo and Database Bar for route /bar. It seems a clever idea to do what is called a “contextual binding”, aka inject instances conditionally based on some runtime value. In Laravel it looks like this: <?php $this->app->when(PhotoController::class) ->needs(Filesystem::class) ->give(function () { return Storage::disk('local'); }); $this->app->when([VideoController::class, UploadController::class]) ->needs(Filesystem::class) ->give(function () { return Storage::disk('s3'); }); This code looks nice and handy at first glance, but in my experience, they are often doing more harm than any good.

Came across this pic from twitter. If I have an adequate number of audience, I would like to make poll out of it. What do you think about this piece of code? A: It is good code. B: It is bad code. I have never done react before (Or I have, but not professional). However this piece of code is very readable to me. If I was tasked with the job to maintain this code, I would appreciate the original author for coding in this fashion.

2020-01-22 日更新：现已提供Hyperf Helm chart。详见repo。 Hyperf官方提供了容器镜像，配置选项又非常开放，将Hyperf部署于云端本身并不复杂。下面我们以Kubernetes为例，对Hyperf默认的骨架包进行一些改造，使它可以优雅的运行于Kubernetes上。本文不是Kubernetes的入门介绍，需要读者已经对Kubernetes有一定了解。 生命周期 容器在Kubernetes上启动以后，Kubernetes会对容器进行两项检查: Liveness Probe和Readiness Probe。Liveness Probe如果没有通过，容器会被重启，而Readiness Probe没有通过，则会暂时将服务从发现列表中移除。当Hyperf作为HTTP Web server启动时，我们只需要添加两条路由就行了。 <?php namespace App\Controller; class HealthCheckController extends AbstractController { public function liveness() { return 'ok'; } public function readiness() { return 'ok'; } } <?php // in config/Routes.php Router::addRoute(['GET', 'HEAD'], '/liveness', 'App\Controller\HealthCheckController@liveness'); Router::addRoute(['GET', 'HEAD'], '/readiness', 'App\Controller\HealthCheckController@readiness'); 在Kubernetes的deployment上配置： livenessProbe:httpGet:path:/livenessport:9501failureThreshold:1periodSeconds:10readinessProbe:httpGet:path:/readinessport:9501failureThreshold:1periodSeconds:10 当然这里我们只是简单了返回‘ok’，显然不能真正检查出健康状况。实际的检查要考虑业务具体场景和业务依赖的资源。例如对于重数据库服务我们可以检查数据库的连接池，如果连接池已满就暂时在Readiness Probe返回状态码503。 服务在Kubernetes销毁时，Kubernetes会先发来SIGTERM信号。进程有terminationGracePeriodSeconds这么长的时间（默认60秒）来自行结束。如果到时间后还没结束，Kubernetes就会发来SIGINT信号来强制杀死进程。Swoole本身是可以正确响应SIGTERM结束服务的，正常情况下不会丢失任何运行中的连接。实际生产中，如果Swoole没有响应SIGTERM退出，很有可能是因为服务端注册的定时器没有被清理。我们可以在OnWorkerExit处清理定时器来保证顺利退出。 <?php // config/autoload/server.php // ... 'callbacks' => [ SwooleEvent::ON_BEFORE_START => [Hyperf\Framework\Bootstrap\ServerStartCallback::class, 'beforeStart'], SwooleEvent::ON_WORKER_START => [Hyperf\Framework\Bootstrap\WorkerStartCallback::class, 'onWorkerStart'], SwooleEvent::ON_PIPE_MESSAGE => [Hyperf\Framework\Bootstrap\PipeMessageCallback::class, 'onPipeMessage'], SwooleEvent::ON_WORKER_EXIT => function () { Swoole\Timer::clearAll(); }, ], // .

The biggest challenge for intermediate developers to understand event loop is that the articles found around the web are either written for die-hard C programmers or simply too long to read. I will give a try on explaining the event loop/epoll(), and stop at a point where interviewers will be sufficiently satisfied. The first thing should be made crystal clear is that things happen on two sides: The kernel and the userspace.

Today we encountered an intriguing production resource leak that deserved to be shared. The Observation: In our production server, the CPU and Memory usage continued to increase while the new request came in. In the development server where no requests were served, the resource usage is stable. The server code was written in Go. Two suspicions were drawn from the observation right away. The problematic code resides in the hot path, ie.

The biggest challenge for intermediate developers to understand event loop is that the articles found around the web are either written for die-hard C programmers or simply too long to read. I will give a try on explaining the event loop/epoll(), and stop at a point where interviewers will be sufficiently satisfied. The first thing should be made crystal clear is that things happen on two sides: The kernel and the userspace.

Today we encountered an intriguing production resource leak that deserved to be shared. The Observation: In our production server, the CPU and Memory usage continued to increase while the new request came in. In the development server where no requests were served, the resource usage is stable. The server code was written in Go. Two suspicions were drawn from the observation right away. The problematic code resides in the hot path, ie.

在Kubernetes中，有时候Pod会异常重启。事后发现的时候错误原因在控制台面板里已经看不到了。 实际上Kubernetes提供了相关的工具。我们可以在不可恢复的异常发生时拦截异常，将其写入/dev/termination-log。 package main import ( "fmt" "os" ) func main() { defer func() { if r := recover(); r != nil { _ = os.WriteFile( "/dev/termination-log", []byte(fmt.Sprintf("panic: %s", r)), os.ModePerm, ) panic(r) } }() // ... } 查看的时候，我们可以在edit pod时在lastState当中找到错误信息： apiVersion:v1kind:Pod...lastState:terminated:containerID:...exitCode:0finishedAt:...message:| panic: goroutine 1 [running]:main.main.func1()/Users/donew/src/kitty/main.go:18+0x131panic(0x5192e20,0x5640e10)/usr/local/Cellar/go/1.16/libexec/src/runtime/panic.go:965+0x1b9main.main()/Users/donew/src/kitty/main.go:21+0x5b... 也可以直接用kubectl 查看。 kubectl get pod termination-demo -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}" 写入的路径可以在terminationMessagePath中更改。 除此之外，如果项目不方便拦截错误，还可以将terminationMessagePolicy设置为FallbackToLogsOnError。 此时使用容器日志输出的最后一块作为终止消息。 日志输出限制为 2048 字节或 80 行，以较小者为准。 实际操作中，FallbackToLogsOnError的长度限制导致有时候会截取太少，丢掉重要信息。 这里提供一个简单的最佳实践：将正常的日志打印到stdout中，将致命错误打印到stderr(Go里的panic默认就是stderr，不用拦截了)，然后将terminationMessagePath设置为/dev/stderr，即可精确的获取Pod重启原因。 参考阅读：https://kubernetes.io/zh/docs/tasks/debug-application-cluster/determine-reason-pod-failure/

The twelve-factor methodology has proven its worth over the years. Since its invention many fields in technology have changed, many among them are shining and exciting. In the age of Kubernetes, service mesh and serverless architectures, the twelve-factor methodology has not faded away, but rather has happened to be a good fit for nearly all of those powerful platforms. Scaffolding a twelve-factor go app may not be a difficult task for experienced engineers, but certainly presents some challenges to juniors.

最近用go搞了一个swoole的边车。是真的边车，用swoole process启动的。挂载到swoole server上跑，swoole起它起，swoole停它停，中间如果go挂了swoole还负责给拉起来。消息投递也照搬swoole task走IPC，从web worker上直接投递，等结果出来再返还web worker。

When we started huijiwiki.com, PHP was the language of the choice. But it was not much a choice. MediaWiki was coded in PHP. It provided us a solid starting point to expand our idea. Since then I have touched many languages and am moderately proficient in some of them. Among them, Go is my current favorite. Rust is the language I want to explore more. Go as well as Rust has a very different nature compared to PHP.

Vue community exploded due to changes in Vue 3. Some comments are full of gunpowder with regards to the recent functional API RFC. Let me show my opinions up front: A. OSS authors clearly deserve more respect. B. I think functional API generally are more expressive compared to the old one. However I would like to provide a counter argument to a very point the defender of RFC made.

最近用go搞了一个swoole的边车。是真的边车，用swoole process启动的。挂载到swoole server上跑，swoole起它起，swoole停它停，中间如果go挂了swoole还负责给拉起来。消息投递也照搬swoole task走IPC，从web worker上直接投递，等结果出来再返还web worker。

One of the first few things I feel strange about gRPC is that gRPC terminate unfinished request based on a deadline mechanism instead of the more common timeout mechanism. In pseudo-code: var timeout = 5 * time.Second; var deadline = time.Now() + 5 * time.Second; As you can see, the deadline mechanism is less straight-forward at first glance. So why bother? I thought this might be another “Google” thing, so I didn’t put my mind to it until recently.

In advertising and other online marketing fields, a common practice is to distribute URLs to the clients so that they can report user action through a simple GET request.

Those parameters are fake. But you got the idea. Longer URLs mean higher bandwidth consumption. Moreover, if your URL is too long, you will risk triggering Error 413: request entity too large. I know nginx throws it.

假设我们有一个监控任务，每5秒钟将监控结果上报。那么问题来了，如果我们现在需要关闭服务，如何保证关闭前最后一批数据不丢失？

有小伙伴抱怨道，WebSocket Server感觉太原始，没有“框架感”。希望Socket.io协议的支持，可以让WebSocket更好用，不再有开篇提到的困惑。

最近用go搞了一个swoole的边车。是真的边车，用swoole process启动的。挂载到swoole server上跑，swoole起它起，swoole停它停，中间如果go挂了swoole还负责给拉起来。消息投递也照搬swoole task走IPC，从web worker上直接投递，等结果出来再返还web worker。

2020-01-22 日更新：现已提供Hyperf Helm chart。详见repo。 Hyperf官方提供了容器镜像，配置选项又非常开放，将Hyperf部署于云端本身并不复杂。下面我们以Kubernetes为例，对Hyperf默认的骨架包进行一些改造，使它可以优雅的运行于Kubernetes上。本文不是Kubernetes的入门介绍，需要读者已经对Kubernetes有一定了解。 生命周期 容器在Kubernetes上启动以后，Kubernetes会对容器进行两项检查: Liveness Probe和Readiness Probe。Liveness Probe如果没有通过，容器会被重启，而Readiness Probe没有通过，则会暂时将服务从发现列表中移除。当Hyperf作为HTTP Web server启动时，我们只需要添加两条路由就行了。 <?php namespace App\Controller; class HealthCheckController extends AbstractController { public function liveness() { return 'ok'; } public function readiness() { return 'ok'; } } <?php // in config/Routes.php Router::addRoute(['GET', 'HEAD'], '/liveness', 'App\Controller\HealthCheckController@liveness'); Router::addRoute(['GET', 'HEAD'], '/readiness', 'App\Controller\HealthCheckController@readiness'); 在Kubernetes的deployment上配置： livenessProbe:httpGet:path:/livenessport:9501failureThreshold:1periodSeconds:10readinessProbe:httpGet:path:/readinessport:9501failureThreshold:1periodSeconds:10 当然这里我们只是简单了返回‘ok’，显然不能真正检查出健康状况。实际的检查要考虑业务具体场景和业务依赖的资源。例如对于重数据库服务我们可以检查数据库的连接池，如果连接池已满就暂时在Readiness Probe返回状态码503。 服务在Kubernetes销毁时，Kubernetes会先发来SIGTERM信号。进程有terminationGracePeriodSeconds这么长的时间（默认60秒）来自行结束。如果到时间后还没结束，Kubernetes就会发来SIGINT信号来强制杀死进程。Swoole本身是可以正确响应SIGTERM结束服务的，正常情况下不会丢失任何运行中的连接。实际生产中，如果Swoole没有响应SIGTERM退出，很有可能是因为服务端注册的定时器没有被清理。我们可以在OnWorkerExit处清理定时器来保证顺利退出。 <?php // config/autoload/server.php // ... 'callbacks' => [ SwooleEvent::ON_BEFORE_START => [Hyperf\Framework\Bootstrap\ServerStartCallback::class, 'beforeStart'], SwooleEvent::ON_WORKER_START => [Hyperf\Framework\Bootstrap\WorkerStartCallback::class, 'onWorkerStart'], SwooleEvent::ON_PIPE_MESSAGE => [Hyperf\Framework\Bootstrap\PipeMessageCallback::class, 'onPipeMessage'], SwooleEvent::ON_WORKER_EXIT => function () { Swoole\Timer::clearAll(); }, ], // .

注解是元编程的一种。元编程从字面意思上说就是编写程序的程序。和普通编程一样，注解在给我们带来便捷的同时，如果使用不当，也有可能造成可读性、可维护性下降等问题。 在某些注解中，可能有很多配置项，比如： //这还不是一个特别夸张的例子 @CircuitBreaker(timeout=0.05, failCounter=1, successCounter=1, fallback="App\Service\UserService::searchFallback") 如果我们的代码里用很多这样复杂的注解，就会引发以下几个问题： 注解中可使用的数据类型表达能力有限，比如必须用方法的字符串全名来表达方法，容易出错。 离开了IDE的帮助，长注解的可读性变得很差。（比如在GitHub上） 同样配置的注解多个地方使用，修改时要改很多地方。 这里我向大家推荐通过继承的方式配置Hyperf内的注解。 下面是一个继承CircuitBreaker（熔断器）注解的例子。 <?php ... /** * @Annotation * @Target({"METHOD"}) * * Shorthand for CircuitBreaker(timeout=0.05, failCounter=1, successCounter=1, fallback="App\Service\UserService::searchFallback") */ class FooCircuitBreakerAnnotation extends CircuitBreakerAnnotation { /** * @var float */ public $timeout = 0.05; /** * @var string */ public $fallback = UserService::class.'::searchFallback'; /** * The counter required to reset to a close state. * @var int */ public $successCounter = 1; /** * The counter required to reset to a open state.

The biggest challenge for intermediate developers to understand event loop is that the articles found around the web are either written for die-hard C programmers or simply too long to read. I will give a try on explaining the event loop/epoll(), and stop at a point where interviewers will be sufficiently satisfied. The first thing should be made crystal clear is that things happen on two sides: The kernel and the userspace.

One of my junior dev was frustrated today. He was supposed to read from a file stream with NodeJS, piping that into the database. However, he got errors thrown everywhere when dealing with larger files. I helped him identified the problem. He had been dumping async calls (the database insertion) faster than the call could be handled. I remotely remember I run into a similar issue years ago when I wrote a web scraper in NodeJS.

在Kubernetes中，有时候Pod会异常重启。事后发现的时候错误原因在控制台面板里已经看不到了。 实际上Kubernetes提供了相关的工具。我们可以在不可恢复的异常发生时拦截异常，将其写入/dev/termination-log。 package main import ( "fmt" "os" ) func main() { defer func() { if r := recover(); r != nil { _ = os.WriteFile( "/dev/termination-log", []byte(fmt.Sprintf("panic: %s", r)), os.ModePerm, ) panic(r) } }() // ... } 查看的时候，我们可以在edit pod时在lastState当中找到错误信息： apiVersion:v1kind:Pod...lastState:terminated:containerID:...exitCode:0finishedAt:...message:| panic: goroutine 1 [running]:main.main.func1()/Users/donew/src/kitty/main.go:18+0x131panic(0x5192e20,0x5640e10)/usr/local/Cellar/go/1.16/libexec/src/runtime/panic.go:965+0x1b9main.main()/Users/donew/src/kitty/main.go:21+0x5b... 也可以直接用kubectl 查看。 kubectl get pod termination-demo -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}" 写入的路径可以在terminationMessagePath中更改。 除此之外，如果项目不方便拦截错误，还可以将terminationMessagePolicy设置为FallbackToLogsOnError。 此时使用容器日志输出的最后一块作为终止消息。 日志输出限制为 2048 字节或 80 行，以较小者为准。 实际操作中，FallbackToLogsOnError的长度限制导致有时候会截取太少，丢掉重要信息。 这里提供一个简单的最佳实践：将正常的日志打印到stdout中，将致命错误打印到stderr(Go里的panic默认就是stderr，不用拦截了)，然后将terminationMessagePath设置为/dev/stderr，即可精确的获取Pod重启原因。 参考阅读：https://kubernetes.io/zh/docs/tasks/debug-application-cluster/determine-reason-pod-failure/

2020-01-22 日更新：现已提供Hyperf Helm chart。详见repo。 Hyperf官方提供了容器镜像，配置选项又非常开放，将Hyperf部署于云端本身并不复杂。下面我们以Kubernetes为例，对Hyperf默认的骨架包进行一些改造，使它可以优雅的运行于Kubernetes上。本文不是Kubernetes的入门介绍，需要读者已经对Kubernetes有一定了解。 生命周期 容器在Kubernetes上启动以后，Kubernetes会对容器进行两项检查: Liveness Probe和Readiness Probe。Liveness Probe如果没有通过，容器会被重启，而Readiness Probe没有通过，则会暂时将服务从发现列表中移除。当Hyperf作为HTTP Web server启动时，我们只需要添加两条路由就行了。 <?php namespace App\Controller; class HealthCheckController extends AbstractController { public function liveness() { return 'ok'; } public function readiness() { return 'ok'; } } <?php // in config/Routes.php Router::addRoute(['GET', 'HEAD'], '/liveness', 'App\Controller\HealthCheckController@liveness'); Router::addRoute(['GET', 'HEAD'], '/readiness', 'App\Controller\HealthCheckController@readiness'); 在Kubernetes的deployment上配置： livenessProbe:httpGet:path:/livenessport:9501failureThreshold:1periodSeconds:10readinessProbe:httpGet:path:/readinessport:9501failureThreshold:1periodSeconds:10 当然这里我们只是简单了返回‘ok’，显然不能真正检查出健康状况。实际的检查要考虑业务具体场景和业务依赖的资源。例如对于重数据库服务我们可以检查数据库的连接池，如果连接池已满就暂时在Readiness Probe返回状态码503。 服务在Kubernetes销毁时，Kubernetes会先发来SIGTERM信号。进程有terminationGracePeriodSeconds这么长的时间（默认60秒）来自行结束。如果到时间后还没结束，Kubernetes就会发来SIGINT信号来强制杀死进程。Swoole本身是可以正确响应SIGTERM结束服务的，正常情况下不会丢失任何运行中的连接。实际生产中，如果Swoole没有响应SIGTERM退出，很有可能是因为服务端注册的定时器没有被清理。我们可以在OnWorkerExit处清理定时器来保证顺利退出。 <?php // config/autoload/server.php // ... 'callbacks' => [ SwooleEvent::ON_BEFORE_START => [Hyperf\Framework\Bootstrap\ServerStartCallback::class, 'beforeStart'], SwooleEvent::ON_WORKER_START => [Hyperf\Framework\Bootstrap\WorkerStartCallback::class, 'onWorkerStart'], SwooleEvent::ON_PIPE_MESSAGE => [Hyperf\Framework\Bootstrap\PipeMessageCallback::class, 'onPipeMessage'], SwooleEvent::ON_WORKER_EXIT => function () { Swoole\Timer::clearAll(); }, ], // .

As the saying goes, Kubernetes is gradually eating the world of software. In the last year, we have switched our on premise infrastructure from Docker Swarm to Kubernetes. Adopting Kubernetes on the cloud is one thing; adopting it on premise and making it production ready is quite another. Many lessons have been learned along our journey. For those who have played with Kubernetes in the vacuum, there are probably some hidden deadly trap yet to be revealed.

Taylor Otwell, the author of Laravel, recently put up a public copy of his unfinished work Laravel Cloud in GitHub. It was taken down by himself briefly after, due to what he described as “too much BS”. Forks are still available everywhere though. Here is my fork. I cloned the repository and skimmed through most of the classes. Here are my takeaways. Dependency Injection is used very lightly. The service container is mostly underfilled.

Sometimes in your daily programming life, you would want to inject different object instances based on the current route or module. For example, you want to connect to Database Foo for route /foo and Database Bar for route /bar. It seems a clever idea to do what is called a “contextual binding”, aka inject instances conditionally based on some runtime value. In Laravel it looks like this: <?php $this->app->when(PhotoController::class) ->needs(Filesystem::class) ->give(function () { return Storage::disk('local'); }); $this->app->when([VideoController::class, UploadController::class]) ->needs(Filesystem::class) ->give(function () { return Storage::disk('s3'); }); This code looks nice and handy at first glance, but in my experience, they are often doing more harm than any good.

One of the first few things I feel strange about gRPC is that gRPC terminate unfinished request based on a deadline mechanism instead of the more common timeout mechanism. In pseudo-code: var timeout = 5 * time.Second; var deadline = time.Now() + 5 * time.Second; As you can see, the deadline mechanism is less straight-forward at first glance. So why bother? I thought this might be another “Google” thing, so I didn’t put my mind to it until recently.

MySQL (InnoDB) transactions are ACID, with A for atomicity. An atomic transaction is an indivisible and irreducible series of database operations such that either all occur, or nothing occurs The atomicity here is different from the atomicity in concurrent programming. An operation acting on shared memory is atomic if it completes in a single step relative to other threads. Suppose you want to read a number from a column, add it by an offset, and write it back to the same column.

I have to admit I am really clumsy when it comes to SQL related stuff. This is the year of 2019, and SQL is not a fancy technology to learn anymore. I managed to get away with not diving deep into SQL for a long time, but the lack of fluency in SQL bites me quite often recently. Yesterday I was forced into a binlog trace of the production MySQL database.

NestJS 6 comes with all-new injection scope support. Namely, instead of enforcing every object managed by DI container to be a singleton, you can now freely assign three types of injection scope to them: singleton, request, and transient. We adopted the request-scoped injection in our project to achieve distributed tracing as well as context-aware logging. The result turned out to be not so great. For starters, the request-scoped injection is way too expensive.

One of my junior dev was frustrated today. He was supposed to read from a file stream with NodeJS, piping that into the database. However, he got errors thrown everywhere when dealing with larger files. I helped him identified the problem. He had been dumping async calls (the database insertion) faster than the call could be handled. I remotely remember I run into a similar issue years ago when I wrote a web scraper in NodeJS.

假设我们有一个监控任务，每5秒钟将监控结果上报。那么问题来了，如果我们现在需要关闭服务，如何保证关闭前最后一批数据不丢失？

最近用go搞了一个swoole的边车。是真的边车，用swoole process启动的。挂载到swoole server上跑，swoole起它起，swoole停它停，中间如果go挂了swoole还负责给拉起来。消息投递也照搬swoole task走IPC，从web worker上直接投递，等结果出来再返还web worker。

2020-01-22 日更新：现已提供Hyperf Helm chart。详见repo。 Hyperf官方提供了容器镜像，配置选项又非常开放，将Hyperf部署于云端本身并不复杂。下面我们以Kubernetes为例，对Hyperf默认的骨架包进行一些改造，使它可以优雅的运行于Kubernetes上。本文不是Kubernetes的入门介绍，需要读者已经对Kubernetes有一定了解。 生命周期 容器在Kubernetes上启动以后，Kubernetes会对容器进行两项检查: Liveness Probe和Readiness Probe。Liveness Probe如果没有通过，容器会被重启，而Readiness Probe没有通过，则会暂时将服务从发现列表中移除。当Hyperf作为HTTP Web server启动时，我们只需要添加两条路由就行了。 <?php namespace App\Controller; class HealthCheckController extends AbstractController { public function liveness() { return 'ok'; } public function readiness() { return 'ok'; } } <?php // in config/Routes.php Router::addRoute(['GET', 'HEAD'], '/liveness', 'App\Controller\HealthCheckController@liveness'); Router::addRoute(['GET', 'HEAD'], '/readiness', 'App\Controller\HealthCheckController@readiness'); 在Kubernetes的deployment上配置： livenessProbe:httpGet:path:/livenessport:9501failureThreshold:1periodSeconds:10readinessProbe:httpGet:path:/readinessport:9501failureThreshold:1periodSeconds:10 当然这里我们只是简单了返回‘ok’，显然不能真正检查出健康状况。实际的检查要考虑业务具体场景和业务依赖的资源。例如对于重数据库服务我们可以检查数据库的连接池，如果连接池已满就暂时在Readiness Probe返回状态码503。 服务在Kubernetes销毁时，Kubernetes会先发来SIGTERM信号。进程有terminationGracePeriodSeconds这么长的时间（默认60秒）来自行结束。如果到时间后还没结束，Kubernetes就会发来SIGINT信号来强制杀死进程。Swoole本身是可以正确响应SIGTERM结束服务的，正常情况下不会丢失任何运行中的连接。实际生产中，如果Swoole没有响应SIGTERM退出，很有可能是因为服务端注册的定时器没有被清理。我们可以在OnWorkerExit处清理定时器来保证顺利退出。 <?php // config/autoload/server.php // ... 'callbacks' => [ SwooleEvent::ON_BEFORE_START => [Hyperf\Framework\Bootstrap\ServerStartCallback::class, 'beforeStart'], SwooleEvent::ON_WORKER_START => [Hyperf\Framework\Bootstrap\WorkerStartCallback::class, 'onWorkerStart'], SwooleEvent::ON_PIPE_MESSAGE => [Hyperf\Framework\Bootstrap\PipeMessageCallback::class, 'onPipeMessage'], SwooleEvent::ON_WORKER_EXIT => function () { Swoole\Timer::clearAll(); }, ], // .

注解是元编程的一种。元编程从字面意思上说就是编写程序的程序。和普通编程一样，注解在给我们带来便捷的同时，如果使用不当，也有可能造成可读性、可维护性下降等问题。 在某些注解中，可能有很多配置项，比如： //这还不是一个特别夸张的例子 @CircuitBreaker(timeout=0.05, failCounter=1, successCounter=1, fallback="App\Service\UserService::searchFallback") 如果我们的代码里用很多这样复杂的注解，就会引发以下几个问题： 注解中可使用的数据类型表达能力有限，比如必须用方法的字符串全名来表达方法，容易出错。 离开了IDE的帮助，长注解的可读性变得很差。（比如在GitHub上） 同样配置的注解多个地方使用，修改时要改很多地方。 这里我向大家推荐通过继承的方式配置Hyperf内的注解。 下面是一个继承CircuitBreaker（熔断器）注解的例子。 <?php ... /** * @Annotation * @Target({"METHOD"}) * * Shorthand for CircuitBreaker(timeout=0.05, failCounter=1, successCounter=1, fallback="App\Service\UserService::searchFallback") */ class FooCircuitBreakerAnnotation extends CircuitBreakerAnnotation { /** * @var float */ public $timeout = 0.05; /** * @var string */ public $fallback = UserService::class.'::searchFallback'; /** * The counter required to reset to a close state. * @var int */ public $successCounter = 1; /** * The counter required to reset to a open state.

The first time I saw P++ my reaction was “meh, it is not going anywhere”. Anything breaks backward compatibility and abandons existing user base will get punished eventually. Python community took a decade to heal from the wound caused by Python3. In short, I believe it is not worth it to trade backward compatibility for language design purity. I also wrote a blog about Vue3 changes on its new functional APIs.

Taylor Otwell, the author of Laravel, recently put up a public copy of his unfinished work Laravel Cloud in GitHub. It was taken down by himself briefly after, due to what he described as “too much BS”. Forks are still available everywhere though. Here is my fork. I cloned the repository and skimmed through most of the classes. Here are my takeaways. Dependency Injection is used very lightly. The service container is mostly underfilled.

Sometimes in your daily programming life, you would want to inject different object instances based on the current route or module. For example, you want to connect to Database Foo for route /foo and Database Bar for route /bar. It seems a clever idea to do what is called a “contextual binding”, aka inject instances conditionally based on some runtime value. In Laravel it looks like this: <?php $this->app->when(PhotoController::class) ->needs(Filesystem::class) ->give(function () { return Storage::disk('local'); }); $this->app->when([VideoController::class, UploadController::class]) ->needs(Filesystem::class) ->give(function () { return Storage::disk('s3'); }); This code looks nice and handy at first glance, but in my experience, they are often doing more harm than any good.

As the job demand is going nowhere in a decade, a lot of new guys want to get into programming. There are numerous courses online, teaching a variety of skills in programming. There are also new shinning frameworks climbing up GitHub trending every day. I see a lot of people, typically newly graduated students, get confused. There seems to be an endless stream of knowledge to learn. How many skills are enough?

When we started huijiwiki.com, PHP was the language of the choice. But it was not much a choice. MediaWiki was coded in PHP. It provided us a solid starting point to expand our idea. Since then I have touched many languages and am moderately proficient in some of them. Among them, Go is my current favorite. Rust is the language I want to explore more. Go as well as Rust has a very different nature compared to PHP.

In advertising and other online marketing fields, a common practice is to distribute URLs to the clients so that they can report user action through a simple GET request.

Those parameters are fake. But you got the idea. Longer URLs mean higher bandwidth consumption. Moreover, if your URL is too long, you will risk triggering Error 413: request entity too large. I know nginx throws it.

DMP stands for Data Management Platform, and per Wikipedia, it is a technology platform used for collecting and managing data, mainly for digital marketing purposes. To put it simply, DMP is a collection of data about user attributes. As you might expect a DMP may contain more than millions of data. The DMP I worked on contained nearly a billion. Open source toolings such as Elasticsearch provided a good average baseline for data analytics, and reasonable response time for general-purpose querying.

One of the first few things I feel strange about gRPC is that gRPC terminate unfinished request based on a deadline mechanism instead of the more common timeout mechanism. In pseudo-code: var timeout = 5 * time.Second; var deadline = time.Now() + 5 * time.Second; As you can see, the deadline mechanism is less straight-forward at first glance. So why bother? I thought this might be another “Google” thing, so I didn’t put my mind to it until recently.

One of my junior dev was frustrated today. He was supposed to read from a file stream with NodeJS, piping that into the database. However, he got errors thrown everywhere when dealing with larger files. I helped him identified the problem. He had been dumping async calls (the database insertion) faster than the call could be handled. I remotely remember I run into a similar issue years ago when I wrote a web scraper in NodeJS.

RxJS is bonkers! Imagine you have an unbounded stream of events. Each has a unique id. Now you want to throttle the stream based on id, ie. each id should not appear more often than X minutes. With RxJS: // Throttle each id in one minute. fromEvent(emitter, 'tick').pipe( groupBy((e: Event) => e.getId()), flatMap(group => group.pipe(throttleTime(60 * 1000))), ).subscribe( //Do your stuff ); Without RxJS: Too much to write.

I have done some quick’n’dirty bid data processing with Apache Beam in past weeks. As someone who try to stay away from JVM, I am not a big data expert at all. However the working with apache beam is a blast (Using JAVA!). The lazy execution style feels right a home. I love wiring a various component into a topology, and only start processing when data arrives. I once heard a Haskell fan said that she thought lazy execution was what set Haskell apart from other programing languages.

Just read a thread in twitter from Matt Klein, where he basically summarized his opinion about complex tooling like service mesh. Though it is not straightforward to me that Matt, the creator of envoy, is actually dissuading people from jumping on the mesh train, his opinion is not really a fresh one. It seems to be a more and more accepted idea that service mesh are not for every body. It solves a domain of problems that only the top of the pyramid would encounter but meanwhile introduces a bunch of other headaches.

One of my junior dev was frustrated today. He was supposed to read from a file stream with NodeJS, piping that into the database. However, he got errors thrown everywhere when dealing with larger files. I helped him identified the problem. He had been dumping async calls (the database insertion) faster than the call could be handled. I remotely remember I run into a similar issue years ago when I wrote a web scraper in NodeJS.

假设我们有一个监控任务，每5秒钟将监控结果上报。那么问题来了，如果我们现在需要关闭服务，如何保证关闭前最后一批数据不丢失？

最近用go搞了一个swoole的边车。是真的边车，用swoole process启动的。挂载到swoole server上跑，swoole起它起，swoole停它停，中间如果go挂了swoole还负责给拉起来。消息投递也照搬swoole task走IPC，从web worker上直接投递，等结果出来再返还web worker。

As the job demand is going nowhere in a decade, a lot of new guys want to get into programming. There are numerous courses online, teaching a variety of skills in programming. There are also new shinning frameworks climbing up GitHub trending every day. I see a lot of people, typically newly graduated students, get confused. There seems to be an endless stream of knowledge to learn. How many skills are enough?

I have done some quick’n’dirty bid data processing with Apache Beam in past weeks. As someone who try to stay away from JVM, I am not a big data expert at all. However the working with apache beam is a blast (Using JAVA!). The lazy execution style feels right a home. I love wiring a various component into a topology, and only start processing when data arrives. I once heard a Haskell fan said that she thought lazy execution was what set Haskell apart from other programing languages.

Came across this pic from twitter. If I have an adequate number of audience, I would like to make poll out of it. What do you think about this piece of code? A: It is good code. B: It is bad code. I have never done react before (Or I have, but not professional). However this piece of code is very readable to me. If I was tasked with the job to maintain this code, I would appreciate the original author for coding in this fashion.

Just read a thread in twitter from Matt Klein, where he basically summarized his opinion about complex tooling like service mesh. Though it is not straightforward to me that Matt, the creator of envoy, is actually dissuading people from jumping on the mesh train, his opinion is not really a fresh one. It seems to be a more and more accepted idea that service mesh are not for every body. It solves a domain of problems that only the top of the pyramid would encounter but meanwhile introduces a bunch of other headaches.

NestJS 6 comes with all-new injection scope support. Namely, instead of enforcing every object managed by DI container to be a singleton, you can now freely assign three types of injection scope to them: singleton, request, and transient. We adopted the request-scoped injection in our project to achieve distributed tracing as well as context-aware logging. The result turned out to be not so great. For starters, the request-scoped injection is way too expensive.

We have carried out the first or maybe last round of rewrite of the Juhui DSP/SSP/ADX with Typescript. A few goals and non-goals have been set for this labor-intensive process. Goals Strong typed. Strong API contracts. Holistic Observability with logging, tracing, and metrics. Layered Architecture and dedicated component. Testcases can be run by anyone. Stateless over stateful. Improved readability. Proper code deprecation process. Non-Goals Use fancy technology.

Vue community exploded due to changes in Vue 3. Some comments are full of gunpowder with regards to the recent functional API RFC. Let me show my opinions up front: A. OSS authors clearly deserve more respect. B. I think functional API generally are more expressive compared to the old one. However I would like to provide a counter argument to a very point the defender of RFC made.

有小伙伴抱怨道，WebSocket Server感觉太原始，没有“框架感”。希望Socket.io协议的支持，可以让WebSocket更好用，不再有开篇提到的困惑。