RSSBOXhttps://blog.imc.re/RSSBOX/favicon.icoSMGoroのRSS订阅 | 一些好康的东西2025-01-10T20:16:13.000Zhttps://blog.imc.re/RSSBOX/SMGoroHexoVocaloid Opera - the End (720p)https://blog.imc.re/RSSBOX/rss/44757927.html2025-01-10T20:16:13.000Z2025-01-10T20:16:13.000Z#1919962 | Vocaloid Opera - The End (720p) | 687.0 MiB | Anime - English-translated | B4A02619AA1FD2A36691D6A2ECEEA07DF46313F4
]]>Highlights From Git 2.48https://blog.imc.re/RSSBOX/rss/86d9f572.html2025-01-10T10:28:50.000Z2025-01-10T10:28:50.000Z
The open source Git project just released Git 2.48 with features and bug fixes from over 93 contributors, 35 of them new. We last caught up with you on the latest in Git back when 2.47 was released.
To celebrate this most recent release, here is GitHub’s look at some of the most interesting features and changes introduced since last time.
Faster SHA-1s without compromising security
When we published our coverage of Git’s 2.47 release, we neglected to mention a handful of performance changes that went in toward the very end of the cycle. Because this version contains a bugfix related to those changes, we figured now was as good a time as any1 to discuss those changes.
You likely know that Git famously uses SHA-1 hashes by default to identify objects within your repository. (We have covered Git’s capability to use SHA-256 as its primary hash function instead, but for this tidbit we’ll focus on Git in its SHA-1 mode). What you may not know is that Git uses SHA-1 hashes internally in a couple of spots, too. Most notable for our purposes is that the pack format includes a trailing SHA-1 that stores the checksum of the preceding bytes. Git uses this data to validate that a pack’s contents matches what was advertised, and didn’t get corrupted in transit.
By default, Git uses a collision detecting implementation of SHA-1, hardening it against common SHA-1 attacks like SHAttered and Shambles. (GitHub also uses the collision detecting SHA-1 implementation). While the collision detecting SHA-1 implementation protects Git against collision attacks, it does so at the cost of a few extra CPU cycles to look for the telltale signs of these attacks while checksumming.
In most cases, the performance impact is negligible, and the benefit outweighs the minor performance cost. But when computing the checksum of a large pack (like when cloning a large repository), the cost adds up. For instance, we used Callgrind and measured that Git spends around 78% of its CPU computing a checksum during a simulated clone of torvalds/linux.
Luckily, the trailing checksum is a data integrity measure, not a security one. For our purposes, this means that we can safely use a faster, non-collision-detecting implementation of SHA-1 specifically when computing trailing checksums (but nowhere else) without compromising security. Git 2.47 introduced new build-time options to specify a separate hash function implementation used specifically when computing trailing checksums. GitHub has used this option, and as a result measured a 10-13% performance improvement in serving fetches/clones across all repositories.
You can try out Git’s ability to select alternative hash function implementations by building with make OPENSSL_SHA1_UNSAFE=1, or other _UNSAFE variants.
Regular readers of this series will no doubt recall our coverage of Git’s range-diff command (introduced back in Git 2.19), and the newer –remerge-diff option (released in Git 2.36). In case you’re a first-time reader, or neither of those ring a bell for you, don’t worry; here’s a brief refresher.
Git’s range-diff command allows you to compare two sequences of commits, including changes to their order, commit messages, and the actual content changes they introduce. This can be useful when comparing a sequence of commits that were rebased (potentially tweaking the order and changes within the patches along the way), to what that set of commits looked like before the rebase.
Git’s --remerge-diff option tells git show, git log, or various diff-related commands to view the differences between where Git would have stopped with the merge, and what is recorded in the merge. This can be useful when dealing with merge conflicts, since the --remerge-diff view will show you the difference between the conflicts and their resolution, showing you how a given merge conflict was handled.
In Git 2.48, these two features meet for the first time, and range-diff now accepts a --remerge-diff option, so that if someone rebases a sequence of commits with --rebase-merges and potentially needs to make some changes, then the changes in merge commits can also be reviewed as part of the range-diff.
As a side effect of this work, a longstanding bug with --remerge-diff was also fixed, which in particular will allow git log –remerge-diff to be used together with options that change the order of commit traversal (such as --reverse).
Beginning all the way back in Git 2.34, the Git project has been focused on reducing memory leaks with the goal of ultimately making Git leak-free. Since Git is a command line tool, each execution typically only lasts for a brief period of time, after which the kernel will free any memory allocated to Git that Git itself did not free. Because of this, memory leaks in Git have not posed a significant practical issue for everyday use.
But, having memory leaks in Git makes it difficult to convert much of Git’s internals into a callable library, where having memory leaks would be a significant issue. To address this, there has been a concerted effort over many years to reduce the number of memory leaks in Git’s codebase, with the ultimate goal of eliminating them altogether.
After much effort toward that end, Git can now run its test suite successfully with leak checking enabled. As a satisfying end result, much of the test infrastructure we talked about back in 2.34 was removed, resulting in a simpler test infrastructure. Making Git memory leak-free represents significant progress toward being able to convert parts of Git’s internals into a callable library.
The Git project uses GNU Make as the primary means to compile Git, meaning that if you can obtain a copy of Git’s source, running make should be all you need to get a working Git binary (provided you have the necessary dependencies, etc.). There are a couple of exceptions, namely that Git has some support for Autoconf and CMake, though they are not as up-to-date as Git’s Makefile.
But as the Git project approaches its 20th anniversary later this year, its Makefile is starting to show its age. There have been over 2,000 commits to the Makefile, resulting in a build script that is nearly 4,000 lines long.
In this release, the Git project has support for a new build system, Meson, as an alternative to building with GNU Make. While support for Meson is not yet as robust as building with Make, Meson does offer a handful of advantages over Make. Meson is easier to use than Make, making the project more accessible to newcomers or contributors who don’t have significant experience working with Make. Meson also has extensive IDE support, and supports out-of-tree and cross-platform builds, which are both significant assets to the Git project.
Git will retain support for Make and CMake in addition to Meson for the foreseeable future, and retain support for Autoconf for a little longer. But if you’re curious to experiment with Git’s newest build system, you can run meson setup build && ninja -C build on a fresh copy of Git 2.48 or newer.
As the Git project has grown over the years, it has accumulated a number of features and modes that, while reasonable when first introduced, have since become outdated or superseded and are now deprecated. In Git 2.48, the Git project began collecting these now-deprecated features in a list stored in Documentation/BreakingChanges.txt.
This document enables the Git project to discuss deprecating certain features and collects the project’s anticipated deprecations in a single place. On the other side of the equation, it allows users to see if they might be affected by an upcoming deprecation, and share their use-case of a particular feature with the project. Check out the list to see if there is anything on there that you might miss, and to get an early picture of what an eventual Git 3.0 release might look like!
If you’ve ever scripted around your repository’s references, you are likely familiar with Git’s for-each-ref command. In case you’re not, for-each-ref is a flexible tool that allows you to list references in your repository, apply custom formatting specifiers to them, and much more.
Back in Git 2.44, we talked about some performance improvements that allowed git for-each-ref to run significantly faster by combining reference filtering and formatting into the same codepath, eliminating the need to store and sort the results in certain conditions.
Git 2.48 extends those changes by allowing us to take advantage of the same optimizations even when asked to output the references in sorted order (under certain conditions). As long as those conditions are met, you can quickly output a small number of references even under --sort=refname independent of how many references your repository actually has.
While we’re on the topic of references, the reftable subsystem has received some more attention in this release. Git’s reftable implementation was updated to avoid explicit dependencies on some of Git’s convenience APIs, making further progress on being able to compile the reftable code without libgit.a. The reftable implementation was also updated to gracefully handle memory allocation failures instead of exiting the process immediately. Last but not least, the reftable code was updated to be able to reuse reference iterators, resulting in faster reference creation and lower memory usage when using reftables.
When you clone from a remote repository, the default branch that the remote repository uses is reflected in refs/remotes/origin/HEAD locally2. In prior versions of Git, subsequent fetches and pulls did not update this symbolic reference. With Git 2.48, if the remote has a default branch but refs/remotes/origin/HEAD is missing locally, then a fetch will update it.
If you want to take it a step further, you can set remote.origin.followRemoteHead configuration to warn or always; if you do so, when refs/remotes/origin/HEAD already exists but does not match the default branch on the remote side, then when you run git fetch it will either warn you about the change or just automatically update refs/remote/origin/HEAD to the appropriate value depending on what setting you used.
Partial clones also received some love this cycle, fixing an infinite loop and avoiding promisor to non-promisor references that could break the repository after a git gc.
游戏诞生之初就和叙事脱不开干系——在电子游戏早期就有了冒险游戏和文字游戏的类别，游戏被当做叙事的载体；而在学术研究中，对于电子游戏的研究，根据对本体的认知更是分化为几种派别，最为突出的就是叙事学取向和游戏学取向。即使是最为激进的游戏学学者，将叙事学研究游戏称为一种“理论帝国主义”，强调游戏的双向动态媒体属性，也无法否认故事、背景故事和情节在游戏中的作用。新媒体学者亨利·詹金斯在2004年的文章《Game Design as Narrative Architecture》（这篇文章在十年后也被引为讨论“环境叙事”的理论）中提到的观点很有代表性：当你回忆起一款游戏，并讲述给其他人的时候，通常是以故事的形式。
在前一部作品《战神4》的结尾，奎托斯与儿子阿特柔斯历经险阻力克强敌，登上了九界最高的山。儿子将母亲的骨灰递给父亲，父亲将骨灰袋推回给了儿子，并说了一句 “We do together, son.（儿子，我们一起）”。到此为止，从剧作角度来说，父子和解的戏剧效果已经达成了。儿子完成了对父亲的理解，父亲也完成了对儿子的认可，这一对父与子的人物关系已经完满了，在续集中要如何进一步深挖这对父子关系，让人物更加完整呢？
《战神5》给出了最摆烂，最不用动脑子的解法。那就是把《战神4》复刻一遍。
所以在《战神5》中，玩家可以看到一个处于青春期事事叛逆的儿子，和一个如同反驳型人格，只会复读‘You’re not ready’的父亲。前期的所有剧情推动全靠父子二人的斗嘴，但这一切都已经在《战神4》里预演了一遍了。两个人的人物形象没有一丝一毫的进步。同时，这也就是“亲情”这种普世的、大众的亲密关系在文艺作品里的流氓之处。试想一下，如果二人不是父子关系或母子关系或兄弟姐妹关系，而是一对普通的队友与搭档，福尔摩斯与华生，蝙蝠侠与罗宾，没头脑与不高兴，观众会像这样不在乎剧情的合理性吗，还会用“父亲对儿子严厉的爱”这样蹩脚的理由来替编剧圆吗？
第二是和人称代词类似的，针对特定人物的称谓。这里我们看DLC中的人物金针骑士蕾妲。她针对不同人使用的都是不同的称谓。例如针对米凯拉，她一直都尊称为“Kindly Miquella”或者“Miquella the Kind”，这里保留原意的话我更倾向于翻译为“仁慈的米凯拉（大人）”。针对梅瑟莫以及其他男性角色，蕾妲使用的称谓是“Sir”，即某某“阁下”或某某“爵士”，在这种语境下我更倾向于认为是“爵士”。而对于托丽娜，蕾妲使用的是“Saint”，即“圣·约翰”“圣·乔治”中的那个“圣”，是带有强烈宗教意味在里的。这些称谓很好地体现了蕾妲是持有严格等级制度观念，怀持着骑士道精神的性格特征，完全可以看做是最后双方意见不合而相互对峙的伏笔。而在中文翻译中，这些独具特色，体现人物性格的称谓，全部被翻译成了某某“大人”。
KRAFTON深度学习部负责人Kangwook Lee负责进行了AI模型“CPC”的发表。他表示“CPC是一种能通过NVIDIA ACE技术构建的游戏专用设备端小型语言模型（On-device SLM for Gaming）与游戏玩家进行互动的新概念角色”，并解释道“CPC不同于现有的NPC（Non Player Character），它的特点是能与玩家进行对话、协作，灵活地理解情况并做出应对” 。
]]>How to Secure Your GitHub Actions Workflows With CodeQLhttps://blog.imc.re/RSSBOX/rss/465b5fb1.html2025-01-09T13:27:11.000Z2025-01-09T13:27:11.000Z
In the last few months, we secured more than 75 GitHub Actions workflows in open source projects, disclosing more than 90 different vulnerabilities. Out of this research, we produced new support for workflows in CodeQL, empowering you to secure yours.
The situation: growing number of insecure workflows
If you have read our series about keeping your GitHub Actions and workflows secure, you already have a good understanding of common vulnerabilities in GitHub Actions and how to solve them.
Unfortunately, we found that these vulnerabilities are still quite common, mostly because of a lack of awareness of how the moving parts interact with each other and what the impact of these vulnerabilities may be for your organization or repository.
To help prevent the introduction of vulnerabilities, identify them in existing workflows, and even fix them using GitHub Copilot Autofix, CodeQL support has been added for GitHub Actions. The new CodeQL packs can be used by code scanning to scan both existing and new workflows. As code scanning and Copilot Autofix are free for OSS repositories, all public GitHub repositories will have access to these new queries, empowering detection and remediation of these vulnerabilities.
In the rest of this post, we’ll see
What we added to our existing CodeQL support, including actions as a first-class language, taint tracking, bash support;
What models and queries we developed;
What vulnerabilities and what new patterns we discovered as part of this work.
Previous attempt
Previously, there was a single CodeQL query capable of identifying simplistic code injections in GitHub workflows. However, this query had several limitations. First, it was bundled with the JavaScript QL packs, meaning users had to enable JavaScript scanning even if they had no JavaScript code in their repositories, which was confusing and misleading. Additionally, the representation of GitHub workflow syntax and grammar was incomplete, making it difficult to express more complex patterns using the existing Abstract Syntax Tree (AST) of GitHub Actions, which is used by static analysis tools such as CodeQL. Most importantly, the CodeQL support for GitHub workflows did not previously include Taint Tracking support and models for non-straightforward sources of untrusted data or dangerous operations.
Taint Tracking is key!
So what is Taint Tracking and how important is it?
Through the previous query for Code Injection, we were able to identify simplistic vulnerabilities such as those cases where a known user-controlled property gets directly interpolated into a Run script:
That was a great starting point, but what about cases such as the following?
The first step of the path is the download of an artifact.
The second and third steps are setting the content of a file from the artifact as the output of the workflow step.
In the last step, the value from the previous step is interpolated in an unsafe manner in a Run script leading to a potential code injection.
In the case above, the source of untrusted data is not simply a GitHub Event Context access of a known untrusted property (for example, github.event.pull_request.body) but rather the download of an artifact. Should all artifacts be considered untrusted? Certainly not. However, in this instance, where the workflow is triggered by a workflow_run event with no branch filters and where an artifact is downloaded from the triggering workflow (github.event.workflow_run.workflow_id), the artifact should be considered untrusted. When decompressed, it may pollute the Runner’s workspace by writing to files in unexpected locations. Consequently, from that step onward, all files in the workspace should be considered untrusted. This example highlights a non-trivial pattern that we need to express using the new actions AST representation to identify sources of untrusted data.
Identifying sources of untrusted data is only the first step towards uncovering more complex injection vulnerabilities. In the example above, it is crucial to understand Bash scripts to determine if they are reading from untrusted files and inserting data into shell variables. It is essential to comprehend how these variables may flow into the output of a step, and, subsequently, how the output flows through different steps, jobs, composite actions, or reusable workflows until they reach a potentially dangerous sink. This understanding is what Taint Tracking and Control Flow will achieve.
In summary, as illustrated in the CodeQL alert above, we can now identify non-obvious sources of untrusted data (for example, git or gh commands or third-party actions) and, more importantly, track this untrusted data throughout complex workflows. These workflows involve multiple steps, jobs, actions, and even entire workflows, allowing us to better understand where this data is used and to report potential vulnerabilities effectively.
Bash support
GitHub’s workflows can execute various scripts, with Bash scripts being among the most common. The new CodeQL packs for GitHub Actions offer basic support for Bash, helping to identify tainted data originating from Bash scripts. For example, commands such as git diff-tree obtain a list of changed files. Tainted data can flow through a script when reading an attacker-controlled file or environment variable into a step’s output or another environment variable. A pertinent example of such a vulnerability could be found in a workflow of Azure CLI repository.
In the alert above, we can see how untrusted data, such as a pull request’s title, is assigned to the TITLE environment variable. This variable is then read and processed by several commands, resulting in a new message variable that gets redirected to the special file pointed to by the GITHUB_ENV variable. A malicious actor could craft a title that results in a multiline message, allowing them to inject arbitrary environment variables into subsequent steps. This, in turn, would enable the attacker to exfiltrate secrets used in the workflow.
The new CodeQL packs are able to parse Bash scripts. While they don’t yet generate a full AST, they already allow us to understand elements such as assignments, pipelines, and redirections, enabling us to report subtle vulnerabilities like the one mentioned above.
Models
As explained in “Keeping your GitHub Actions and workflows secure Part 2: Untrusted input,” GitHub’s event context is the most common source of untrusted data. Properties such as github.event.issue.title, github.event.pull_request.head.ref, or github.event.comment.body are typical sources of untrusted data. However, any third-party action may introduce untrusted data. For instance, an action that returns a list of filenames changed in a pull request should be considered a source of untrusted data. Similarly, actions that parse an issue body or comment for a command or structured data should also be treated as sources of untrusted data.
The same applies to actions that pass data from one of their inputs to their outputs or into an environment variable, therefore acting as taint steps (summaries). Actions such as actions/github-script, azure/cli, or azure/powershell should be considered sinks for Code Injection, similar to a Run’s step.
We have analyzed thousands of popular third-party actions and identified a number of models now incorporated into the analysis:
62 sources
129 summaries
2199 sinks
Queries
The previous support for GitHub Actions contained a single query for code injection, whereas the new CodeQL packs incorporate 18 new queries, including the Code Injection and Environment Variable Injection queries mentioned above.
Execution of Untrusted Code
Execution of Untrusted Code (TOCTOU)
Artifact Poisoning
Code Injection
Environment Variable Injection
Path Injection
Unpinned Action Tag
Improper Access Control
Excessive Secrets Exposure
Secrets In Artifacts
Expression is Always True
Unmasked Secret Exposure
Cache Poisoning (Code Injection, Direct Cache, Untrusted Code)
Use of Known Vulnerable Actions
Missing Action Permissions
Argument Injection (Experimental)
Code Execution on Self Hosted Runners (Experimental)
Output Clobbering (Experimental)
Results
For the past few months, we have been testing the new queries on thousands of open source projects to validate their accuracy and performance. The results have been very impressive, allowing us to identify and report vulnerabilities in numerous critical organizations and repositories, such as Microsoft, Azure, GitHub, Eclipse, Jupyter, Adobe, AWS, Cloudflare, Discord, Hibernate, HuggingFace, and Apache.
The table below shows all the repositories affected, along with their GitHub stars, to give an idea of the impact that a supply chain attack could have had in these projects:
Repository
Stars
ant-design/ant-design
92,412
Excalidraw/excalidraw
84,021
apache/superset
62,589
withastro/astro
46,604
Stirling-Tools/Stirling-PDF
44,988
geekan/MetaGPT
44,901
Kong/kong
39,221
LAION-AI/Open-Assistant
37,045
appsmithorg/appsmith
34,352
gradio-app/gradio
33,709
DIYgod/RSSHub
33,432
calcom/cal.com
32,282
milvus-io/milvus
30,299
k3s-io/k3s
28,010
discordjs/discord.js
25,390
element-plus/element-plus
24,488
cilium/cilium
20,150
monkeytypegame/monkeytype
15,635
amplication/amplication
15,196
docker-mailserver/docker-mailserver
14,643
jupyterlab/jupyterlab
14,167
openimsdk/open-im-server
14,041
quarkusio/quarkus
13,771
espressif/arduino-esp32
13,609
sympy/sympy
12,967
ionic-team/stencil
12,561
zephyrproject-rtos/zephyr
10,819
qgis/QGIS
10,569
trinodb/trino
10,413
OpenFeign/feign
9,490
marimo-team/marimo
7,583
dream-num/univer
7,021
aws/karpenter-provider-aws
6,782
hibernate/hibernate-orm
5,976
ant-design-blazor/ant-design-blazor
5,809
litestar-org/litestar
5,511
New vulnerability patterns
Having triaged and reported numerous alerts, we have identified some common patterns that often lead to vulnerabilities in GitHub workflows:
Misuse of pull_request_target trigger
The pull_request_target event trigger, while offering powerful automation capabilities in GitHub Actions, harbors a dark side filled with potential security pitfalls. This event trigger, designed to execute workflows within the context of pull request’s base branch, presents special characteristics that severely increase the impact in case of any vulnerability. A workflow activated by pull_request_target and triggered from a fork operates with significant privileges, in contrast to the pull_request event:
It is able to read repository and organization secrets.
It is allowed to have write permissions.
It circumvents the usual safeguards of pull request approvals, allowing workflows to run unimpeded even if approval mechanisms are configured for standard pull_request events.
It normally runs in the context of the default branch, which, as we will see, may allow malicious actors to poison the action’s cache and move laterally to other, more privileged workflows even when removing all permissions to the vulnerable workflow.
When working with pull_request_triggered workflows, we have to be very careful and pay special attention to the following scenarios:
Code execution from untrusted sources: The ability to execute code from forked pull requests is a double-edged sword. A malicious actor can submit a seemingly innocuous pull request that, when triggered by pull_request_target, unleashes havoc. This malicious code, running in the context of the target repository’s environment, could exfiltrate secrets or even tamper with repository contents and releases. The danger lies in the inadvertent checkout and execution of code from untrusted sources. Common mistakes include using the actions/checkout action with a reference to the pull request’s head branch.
Time of check to time of use (TOCTOU) attacks. Even with approval requirements in place (such as requiring the pull request to have a specific label that attackers are not able to set on their own), attackers can leverage the element of time to bypass security measures. A malicious actor can submit a seemingly harmless pull request, patiently wait for approval, and then swiftly update the pull request with malicious code before the workflow execution. The workflow, relying on a mutable reference like a branch name, falls prey to this “time of check to time of use” (TOCTOU) attack, unwittingly executing the newly injected malicious code. Confusion surrounding context variables, specifically head.ref and head.sha, can lead to vulnerabilities. head.ref, pointing to the branch, is susceptible to manipulation by attackers. In contrast, head.sha, referencing the specific commit, provides a reliable and immutable pointer to the reviewed and approved code. Using the incorrect variable can create an opening for attackers to inject malicious code after approval.
Lurking threats in non-default branches. Vulnerabilities often linger in non-default branches, even after being addressed in the default branch. An attacker can target these vulnerable versions of the workflows residing in non-default branches, exploiting them by submitting pull requests specifically to those branches.
Cache poisoning. As a mitigation, developers may strip out all read and write permissions from these workflows when in need to run untrusted code. However, the seemingly innocuous permissions: {} configuration can unexpectedly pave the way for cache poisoning attacks. This attack vector involves injecting malicious content into the cache, which can subsequently affect other workflows relying on the poisoned cache entries. Even if a workflow doesn’t have write access to the repository, it can still poison the cache, leading to potential code execution or data manipulation in other workflows that utilize the compromised cache.
If we really need to use this trigger event, there are a few ways to harden the workflows to prevent any abuses:
Repository checks. Implementing stringent repository checks is crucial for thwarting attacks originating from forked pull requests. One effective method is to configure workflows to execute only for pull requests originating from the base repository, effectively blocking any attempts from external forks. This can be achieved by using conditions such as github.event.pull_request.head.repo.owner.login == “myorg” to restrict workflow execution.
Actor checks. Verifying the permissions of the pull request author is another layer of defense. By restricting workflow execution to trusted actors, such as members of the organization or approved collaborators, the risk of malicious code injection from unauthorized sources can be significantly reduced. Never use a hardcoded list of user names, since the users may lose the permissions with time or the user names may be left abandoned for an attacker to claim.
Workflow splitting. The above mitigations may not be useful if we need to run a workflow for forks or arbitrary users. In those cases, splitting workflows into unprivileged and privileged components is a powerful security strategy. Unprivileged workflows, triggered by pull_request events, handle the initial processing of pull requests without access to sensitive secrets or write permissions. Privileged workflows, activated by workflow_run events, are invoked only after the unprivileged workflow has completed its checks. This separation ensures that potentially malicious code from forked pull requests never executes within a privileged context. The unprivileged workflow will generally need to communicate and pass information to the privileged one. This is a crucial security boundary, and, as we will see in the next section, any data coming from the unprivileged workflow should be considered untrusted and potentially dangerous.
Security boundaries and workflow_run event
The workflow_run event trigger in GitHub Actions is designed to automate tasks based on the execution or completion of another workflow. It may grant write permissions and access to secrets even if the triggering workflow doesn’t have such privileges. While this is beneficial for tasks like labeling pull requests based on test results, it poses significant security risks if not used carefully.
The workflow_run trigger poses a risk because it can often be initiated by an attacker. Some maintainers were surprised by this, believing that their triggering workflows, which were run on events such as release, were safe. This assumption was based on the idea that since an attacker couldn’t trigger a new release, they shouldn’t be able to initiate the triggering workflow or the subsequent workflow_run workflow.
The reality is that an attacker can submit a pull request that modifies the triggering workflow and even replace the triggering events. Since pull_request workflows run in the context of the pull request’s HEAD branch, the modified workflow will run and, upon completion, will be able to trigger an existing workflow_run workflow. The danger arises from the fact that even if the triggering pull_request workflow is not privileged, the triggered workflow_run workflow will have access to secrets and write-scoped tokens, even if the initial workflow did not have those privileges. This enables privilege escalation attacks, allowing attackers to execute malicious code with elevated permissions within the CI/CD pipeline.
Another significant pitfall with the workflow_run event trigger is artifact poisoning. Artifacts are files generated during a workflow run that can be shared with other workflows. Attackers can poison these artifacts by uploading malicious content through a pull request. When a workflow_run workflow downloads and uses these poisoned artifacts, it can lead to arbitrary code execution or other malicious activities within the privileged workflow. The issue is that many workflow_run workflows do not verify the contents of downloaded artifacts before using them, making them vulnerable to various attacks.
Securing workflow_run workflows requires a multi-faceted approach. By understanding the inherent risks and implementing the recommended mitigations, developers can leverage the automation benefits of workflow_run while minimizing the potential for security compromises.
Effective mitigations
Limit workflow scope with branch filters: Specify the branches where workflow_run workflows can be triggered using the branches filter. This helps restrict the scope of potential attacks by preventing them from being triggered on branches from forks.
Verify event origin: Incorporate a check like github.event_name != 'pull_request' to prevent workflow_run workflows from being triggered by pull requests from forks. This adds an extra layer of protection by ensuring that the triggering workflow originates from a trusted source.
Treat artifacts as untrusted: Treat all downloaded artifacts as potentially malicious and implement rigorous validation checks before using them. Always unzip artifacts to a temporary directory like /tmp to prevent potential file overwrites, that is, the pollution of the workspace.
Avoid defining environment variables: Minimize the use of environment variables, especially when handling untrusted data. Environment variables can be vulnerable to injection attacks, potentially allowing attackers to modify their values and execute malicious code.
Handle output variables with caution: Exercise caution when defining output variables from artifact’s content, as they can also be vulnerable to manipulation by attackers. Always validate the contents of output variables (for example, that it is a number, not a string) before using them in subsequent steps or other workflows.
Non-effective mitigations
Repository checks: We found several workflows (for example, AWS Karpenter Provider, or Cloudflare Workers SDK) relying solely on repository checks, such as verifying the repository owner (github.repository_owner == 'myorg'), which is not effective in mitigating workflow_run risks since the workflow always runs in the context of the default branch which belongs to the organization.
IssueOops: Security pitfalls with issue_comment trigger
The issue_comment event trigger in GitHub Actions is a powerful tool for automating workflows based on comments on issues and pull requests. When applied in the context of IssueOps, it can streamline tasks like running commands in response to specific comments. However, this convenience comes with significant security risks that must be carefully considered.
TOCTOU vulnerabilities: Similar to the pull_request_target event trigger, workflows using issue_comment can be vulnerable to TOCTOU attacks. If the workflow checks out and executes code from a pull request based on an issue comment, an attacker could exploit the time window between the comment and the workflow execution. An attacker might initially submit a harmless pull request, waiting for an administrator to review and approve the workflow by adding a comment. Once the approval is given, the attacker could quickly update the pull request with malicious code, which would then be executed by the workflow.
Bypassing pull request approval mechanisms: The issue_comment event trigger is not subject to the pull request approval mechanisms intended to prevent abuse. Even if the workflows triggered by pull request require approvals, an attacker can trigger an issue_comment workflow by simply adding a comment to the pull request, potentially executing malicious code without any review.
Mitigating the risks
Shifting to label gates: Instead of relying on issue comments to trigger critical workflows, consider adopting a label gates approach. Label gates use labels to trigger specific actions, allowing for more granular control and better security. Since the labeled activity type for a pull_request trigger contains details about the latest commit SHA of the pull request, there is no need for workflow to resolve a pull request number into the latest commit SHA, as it is the case with issue_comment, and, therefore, there is no window for an attacker to modify the pull request. Remember to use the commit SHA rather than the HEAD reference to prevent TOCTOU vulnerabilities.
Ineffective or incomplete mitigations
Actor checks: Relying solely on actor checks (verifying the identity or permissions of the commenter) is ineffective. The actor triggering the workflow might not be the same one trying to exploit it, further rendering actor checks unreliable. This is the case for TOCTOU vulnerabilities where an attacker can submit a legitimate pull request and wait for an admin to trigger an IssueOp and then swiftly mutate the pull request by adding a new commit with malicious code on it.
Date checks: Comparing timestamps to verify that the last commit occurred before the triggering comment is also unreliable. Currently, GitHub has no reliable way to figure out the date when a commit was pushed to a repository. An attacker can forge commit dates, rendering these checks useless in preventing malicious code execution.
Repository checks: Verifying the origin repository is not a useful mitigation for issue_comment event triggers. The issue_comment event always executes within the context of the target repository’s default branch, making repository checks redundant.
Wrapping up
The new CodeQL support for GitHub Actions is in public preview. The new QL packs allow you to scan your repository for a variety of vulnerabilities in GitHub Actions, helping prevent supply chain attacks in the OSS software we all depend on! If you want to give them a try, take one of the following steps depending on your case:
Your repository is newly configured for default setup: Code scanning will automatically attempt to analyze actions if any workflows are present on the default branch. You have nothing to do; keep calm and fix the potential alerts.
Your repository is already configured for default setup: You need to edit the Default Setup settings and explicitly enable actions analysis.
Your repository is using advanced setup: You just have to add actions to the language matrix.
]]>How to Use GitHub Copilot: What It Can Do and Real-World Exampleshttps://blog.imc.re/RSSBOX/rss/8927e0cb.html2025-01-09T07:33:16.000Z2025-01-09T07:33:16.000Z
Since the free version of GitHub Copilot launched last month, you’ve asked lots of questions, like “Is it free for everyone?” (Yes!), “Can Copilot make changes to multiple files?” (Yes again!), “What’s the name of Copilot’s mascot?” (It doesn’t have a name…yet.), and most of all: “What can GitHub Copilot actually do?”
The short answer is a lot. The longer answer covers a bunch of different use cases, workflows, coding languages, and much more, so it’s a good thing we have some space to talk about it!
We’ll update this list as GitHub Copilot continues to evolve and we add new, awesome features like we’ve been doing since we first launched it a few years ago.
Ready? Let’s go.
What is GitHub Copilot?
GitHub Copilot is a generative AI tool that functions as a coding assistant to help you write code faster and so much more.
One of the main advantages of Copilot is it draws context from your coding environment, open tabs, and your GitHub projects (including your pull requests, issues, discussions, and codebase). For example, you can ask Copilot Chat to provide a summary of code in a given repository or explain how pieces of code in your app work. In either case, it will provide an answer with the contextual knowledge of your project. You can also ask Copilot to help document, debug, and refactor code—all without the need to context switch or copy over large sections of code to another application.
GitHub Copilot is available for anyone to use with GitHub Copilot Free. If you want more advanced Copilot capabilities, we offer Pro, Business, and Enterprise tiers, too. With these options, you can pick the solution that works best for you. For your reference, we’ve included a section on the different access tiers.
How are developers using GitHub Copilot?
As GitHub Copilot capabilities continue to grow, developers are using it in a number of ways including:
Getting code completion suggestions while they type in the IDE
Asking Copilot Chat to explain how a section of code works
Generating tests and helping to fix code that fails a given test
Migrating code to a different language
Refactoring existing code
Explaining code you’re working with
Using Copilot Extensions to work with other applications and tools in your tech stack (you can even build a custom one)
Generating documentation to describe a set of changes in a pull request
What GitHub Copilot can do (and how to use it)
So, how can GitHub Copilot help you in your projects? Let’s jump in.
What’s new in GitHub Copilot
This is a living document, so we’ll update this section with the latest and greatest from Copilot as it continues to evolve.
Choose the AI model that powers GitHub Copilot
With GitHub Copilot, you can choose the model you want to work with. In Copilot Free you have access to Anthropic’s Claude 3.5 Sonnet and OpenAI’s GPT-4o models. And if you use one of the Copilot paid tiers, you’ll have more models to choose from to customize your experience.
In-line code completion
When coding, GitHub Copilot can offer coding suggestions directly in your coding environment. It can both provide code to complete what you’re currently working on or respond to natural language prompts to generate code. Developers often use this for generating boilerplate code and common coding snippets—such as the getter and setter functions of variables.
But Copilot isn’t limited to simple and repetitive code completion. Using Copilot Chat, you can tell it what you want it to do by describing the task in plain language or the /new slash command, it will then provide some code suggestions to help you get started.
💡 Pro tip: Getting the most out of this feature requires you to craft good prompts.You can learn more about effective prompting with GitHub Copilot in our documentation on prompt engineering.
Generate unit tests
Writing unit tests requires knowing the limits, edge cases, and what might cause failures in your code. Since Copilot knows about your project on GitHub and what you’re working on in your IDE, it can create unit tests to help you.
To do this, select a given block of code. Then either right-click and use the context menu or the /tests slash command. Copilot will then analyze your code and try to provide suggestions for valid inputs, edge cases, and invalid inputs.
You can tap into GitHub Copilot’s knowledge of coding patterns to ask it to help explain how a block of code works. This is particularly useful if you’re onboarding to a new project, especially if it’s in a language you don’t have a lot of experience in.
To use this feature, select some code and use either the context menu or the /explain slash command. Copilot will then explain what the code is doing—and then generate documentation on it with another prompt.
This isn’t just useful for onboarding or making sure you understand a piece of code. You can also use this as a building block for other tasks such as helping to translate code from one language to another.
If you’re tasked with upgrading a codebase to a new version of a framework, it’s not as easy as recompiling the project in the new version. That often results in several errors that you need to track down and resolve. GitHub Copilot can help you here.
You can just ask why your code is producing errors. From there, you can have Copilot provide suggestions on how to fix your code to make it compliant with the new version of the framework. Then ask Copilot for suggestions on how to update the code to fit in a new framework.
Explain a codebase (or a repository on GitHub)
Trying to understand a new codebase? Ask Copilot what a repository does or to summarize key components and it will provide a summary. To get started, click the Copilot icon in the top-right corner of your screen and ask a question. You can also prompt Copilot to debug your workflows with GitHub Actions. If you want to focus Copilot’s attention, you can highlight individual files or folders in a repository, too.
Let’s say you need to migrate legacy software from COBOL to a more modern coding language. GitHub Copilot can help you with this, even if you don’t have a lot of experience with the source language.
First, ask it to explain what some code is doing, and then use that explanation to create code in the destination language that performs the same functionality. You can also ask it to provide suggestions on how to write the code in the destination language.
Closely tied to the idea of explaining how code works, you can ask GitHub Copilot to generate documentation for sections of code or entire projects. This can be incredibly useful, and a huge time saver.
In Copilot Free, you can also ask it to provide a summary of given code blocks within your IDE by using the slash command /docs. In the paid versions, you can also generate documentation from issues and pull requests.
One real-world example of this involves legacy COBOL-based applications. You can ask Copilot to help provide the missing docs for a given project. (Visit documenting legacy code to get started).
Debugging code
We’ve all been there—you have a section of code that’s spitting out an error and you aren’t sure how to fix it. Sometimes you try to implement a fix, and then you get more errors (classic 🤦).
To get help from GitHub Copilot, highlight the code in question and either right-click and select “Copilot” or use the /fix slash command in Copilot Chat. You can also ask Copilot Chat why something isn’t working to start a conversation about what the problem might be.
Copilot works by analyzing your code and drawing upon its training data. It also looks at the context of your larger project to identify patterns that might be causing the problem.
Want to make your code cleaner? Perform better? Same. This is a place where GitHub Copilot can help—and I’m speaking from experience. Just highlight a specific code block (or point Copilot at a specific file with the chat variable command #file) and ask Copilot for suggestions to improve your code.
Copilot will then search for specific patterns and ways to refactor your code. For example, it might recommend breaking up a long method with multiple tasks or avoiding nested logic checks to improve readability or performance. You can quickly review and accept suggestions to refactor your code—all right in your IDE.
Copilot Edits provide the functionality for you to edit multiple files at once. This means you don’t need to manually open each file and insert the changes—which is a huge time saver. It also helps reduce the possibility of making mistakes when making manual changes.
You can access this feature through the Copilot Chat menu.
Once you select Open Copilot Edits, prompt Copilot with the changes you want to make. It will then determine which files in your working set need to be updated based on your prompt and provide a description of the changes it recommends for each file. You can then review the changes and individually accept or reject them.
💡 Pro tip: Right now, this feature is only available in VS Code.
When using GitHub Copilot, you can create a file that will provide custom instructions that will be included in all Copilot queries. This lets you reuse some of the same context without needing to manually enter it in each prompt.
When creating prompts, offering more details helps Copilot provide more relevant responses (e.g. related to the tools you use or how your team works). Including some of the same information in each prompt once you learn what works can be very helpful. By placing these persistent details in a file, you can tailor the way GitHub Copilot works to deliver higher quality responses.
💡 Pro tip: Right now, this is only available with Copilot Chat through VS Code and Visual Studio.
Breaking down the different GitHub Copilot tiers from free to enterprise
Currently, everyone has access to the GitHub Copilot Free tier. You can start using GitHub Copilot Free in VS Code by authorizing the extension or through github.com by clicking the Copilot icon in the top right-hand side of the interface.
For more information, see this video about how to set up GitHub Copilot Free.
We also offer additional tiers with more capabilities via Pro, Business, and Enterprise with more advanced capabilities, depending on your needs.
We hope this answers your questions about Copilot’s capabilities and inspires you to give it a try in your next project. Here’s a recap of the resources we shared throughout this article:.
]]>Open Source All the Way Down: Upgrading Our Developer Documentationhttps://blog.imc.re/RSSBOX/rss/dea977e3.html2025-01-08T14:00:00.000Z2025-01-08T14:00:00.000Z
At Cloudflare, we treat developer content like a product, where we take the user and their feedback into consideration. We are constantly iterating, testing, analyzing, and refining content. Inspired by agile practices, treating developer content like an open source product means we approach our documentation the same way an open source software project is created and maintained. Open source documentation empowers the developer community because it allows anyone, anywhere, to contribute content. By making both the content and the framework of the documentation site publicly accessible, we provide developers with the opportunity to not only improve the material itself but also understand and engage with the processes that govern how the documentation is built, approved, and maintained. This transparency fosters collaboration, learning, and innovation, enabling developers to contribute their expertise and learn from others in a shared, open environment. We also provide feedback to other open source products and plugins, giving back to the same community that supports us.
Building the best open source documentation experience
Great documentation empowers users to be successful with a new product as quickly as possible, showing them how to use the product and describing its benefits. Relevant, timely, and accurate content can save frustration, time, and money. Open source documentation adds a few more benefits, including building inclusive and supportive communities that help reduce the learning curve. We love being open source!
While the Cloudflare content team has scaled to deliver documentation alongside product launches, the open source documentation site itself was not scaling well. developers.cloudflare.com had outgrown the workflow for contributors, plus we were missing out on all the neat stuff created by developers in the community.
Just like a software product evaluation, we reviewed our business needs. We asked ourselves if remaining open source was appropriate? Were there other tools we wanted to use? What benefits did we want to see in a year or in five years? Our biggest limitations in addition to the contributor workflow challenges seemed to be around scalability and high maintenance costs for user experience improvements.
After compiling our wishlist of new features to implement, we reaffirmed our commitment to open source. We valued the benefit of open source in both the content and the underlying framework of our documentation site. This commitment goes beyond technical considerations, because it's a fundamental aspect of our relationship with our community and our philosophy of transparency and collaboration. While the choice of an open source framework to build the site on might not be visible to many visitors, we recognized its significance for our community of developers and contributors. Our decision-making process was heavily influenced by two primary factors: first, whether the update would enhance the collaborative ecosystem, and second, how it would improve the overall documentation experience. This focus reflects that our open source principles, applied to both content and infrastructure, are essential for fostering innovation, ensuring quality through peer review, and building a more engaged and empowered user community.
Cloudflare developer documentation: A collaborative open source approach
Cloudflare’s developer documentation is open source on GitHub, with content supporting all of Cloudflare’s products. The underlying documentation engine has gone through a few iterations, with the first version of the site released in 2020. That first version provided dev-friendly features such as dark mode and proper code syntax.
2021 update: enhanced documentation engine
In 2021, we introduced a new custom documentation engine, bringing significant improvements to the Cloudflare content experience. The benefits of the Gatsby to Hugo migration included:
Faster development flow: The development flow replicated production behavior, increasing iteration speed and confidence. Preview links via Cloudflare Pages were also introduced, so the content team and stakeholders could quickly review what content would look like in production.
Custom components: Introduced features like resources-by-selector which let us reference content throughout the repository and gave us the flexibility to expand checks and automations.
Structured changelog management: Implementation of structured YAML changelog entries which facilitated sharing with various platforms like RSS feeds, Developer Discord, and within the docs themselves.
Improved performance: Significant page load time improvements with the migration to HTML-first and almost instantaneous local builds.
These features were non-negotiable as part of our evaluation of whether to migrate. We knew that any update to the site had to maintain the functionality we’d established as core parts of the new experience.
2024 update: Say “hello, world!” to our new developer documentation, powered by Astro
After careful evaluation, we chose to migrate from Hugo to the Astro (and by extension, JavaScript) ecosystem. Astro fulfilled many items on our wishlist including:
Enhanced content organization: Improved tagging and better cross-referencing of related pages.
Extensibility: Support for user plugins like starlight-image-zoom for lightbox functionality.
Development experience: Type-checking at build time with astro check, along with syntax highlighting, Intellisense, diagnostic messages, and plugins for ESLint, Stylelint, and Prettier.
JavaScript/TypeScript support: Aligned the docs site framework with the preferred languages of many contributors, facilitating easier contribution.
CSS management: Introduction of Tailwind and scoped styles.
Starlight, Astro’s documentation theme, was a key factor in the decision. Its powerful component overrides and plugins system allowed us to leverage built-in components and base styling.
How we migrated to Astro
Content needed to be migrated quickly. With dozens of pull requests opened and merged each day, entering a code freeze for a week simply wasn’t feasible. This is where the nature of abstract syntax trees (ASTs) came into play, only parsing the structure of a Markdown document rather than details like whitespace or indentation that would make a regular expression approach tricky.
With Hugo in 2021, we configured code block functionality like titles or line highlights with front matter inside the code block.
if (title) { node.meta = `title="${title}"`; } if (highlight) { node.meta += ` {${highlight}}`; } node.value = body; return;}
})
The migration in numbers
When we migrated from Gatsby to Hugo in 2021, the pull request included 4,850 files and the migration took close to three weeks from planning to implementation. This time around, the migration was nearly twice as large, with 8,060 files changed. Our planning and migration took six weeks in total:
10 days: Evaluate platforms, vendors, and features
14 days: Migrate the components required by the documentation site
The migration resulted in removing a net -19,624 lines of code from our maintenance burden.
While the number of files had grown substantially since our last major migration, our strategy was very similar to the 2021 migration. We used Markdown AST and astray, a utility to walk ASTs, created specifically for the previous migration!
What we learned
A website migration like our move to Astro/Starlight is a complex process that requires time to plan, review, and coordinate, and our preparation paid off! Including our Cloudflare Community MVPs as part of the planning and review period proved incredibly helpful. They provided great guidance and feedback as we planned for the migration. We only needed one day of code freeze, and there were no rollbacks or major incidents. Visitors to the site never experienced downtime, and overall the migration was a major success.
During testing, we ran into several use cases that warranted using experimental Astro APIs. These APIs were always well documented, thanks to fantastic open source content from the Astro community. We were able to implement them quickly without impacting our release timeline.
We also ran into an edge case with build time performance due to the number of pages on our site (4000+). The Astro team was quick to triage the problem and begin investigation for a permanent fix. Their fast, helpful fixes made us truly grateful for the support from the Astro Discord server. A big thank you to the Astro/Starlight community!
Contribute to developers.cloudflare.com!
Migrating developers.cloudflare.com to Astro/Starlight is just one example of the ways we prioritize world-class documentation and user experiences at Cloudflare. Our deep investment in documentation makes this a great place to work for technical writers, UX strategists, and many other content creators. Since adopting a content like a product strategy in 2021, we have evolved to better serve the open source community by focusing on inclusivity and transparency, which ultimately leads to happier Cloudflare users.
We invite everyone to connect with us and explore these exciting new updates. Feel free to reach out if you’d like to speak with someone on the content team or share feedback about our documentation. You can share your thoughts or submit a pull request directly on the cloudflare-docs repository in GitHub.
]]>Building LATAM’s Future Tech Workforce With AIhttps://blog.imc.re/RSSBOX/rss/868f80a5.html2025-01-07T08:34:05.000Z2025-01-07T08:34:05.000ZRead this post in Spanish
Estamos emocionados de celebrar el tercer año de Git Commit, nuestro programa anual dedicado a cerrar la brecha de habilidades tecnológicas y empoderar a estudiantes de todo el país. Este año, también nos enorgullece lanzar Git Commit 101, nuestro primer curso gratuito de IA en línea en español, disponible para estudiantes y educadores durante todo el año.
¿Por qué Uruguay?
Con una de las tasas más altas de conectividad a internet en América Latina (más del 70%), lidera la región en transformación digital y educación. Su compromiso con la innovación se refleja en su clasificación entre las cinco principales economías latinoamericanas en el Índice Global de Innovación, destacándose en producción de conocimiento y desarrollo de capital humano.
Programas como Git Commit prosperan en este entorno, alineándose perfectamente con la misión del país de construir una economía basada en el conocimiento y crear oportunidades para comunidades desfavorecidas.
Presentamos Git Commit 101
Este año, en colaboración con Microsoft AI for Good, lanzamos Git Commit 101, un curso gratuito de IA en línea transmitido en vivo desde el AI Co-Innovation Lab de Microsoft en Montevideo, el único laboratorio de este tipo en el hemisferio sur. El curso introduce a los estudiantes a los fundamentos de la IA, herramientas de GitHub y GitHub Copilot, brindándoles habilidades que pueden aplicar de inmediato. En sus primeras semanas, el curso en línea ya ha atraído a más de 20,000 espectadores.
Nuestro evento presencial de Git Commit se llevó a cabo en Centros Educativos Impulso y la Universidad ORT. Juntos recibimos a 125 estudiantes, muchos de ellos de contextos vulnerables, provenientes de 13 instituciones participantes. Los estudiantes escucharon a oradores inspiradores, incluyendo al Dr. Juan M. Lavista Ferres, Vicepresidente Corporativo y Científico Principal de Datos del AI for Good Lab de Microsoft; Alejandro Pazos, Gerente País de Microsoft en Uruguay; Eliane Elbaum, Gerente del AI Co-Innovation Lab de Microsoft; y Mario Rodriguez, Director de Producto de GitHub.
“Es muy importante que cada uno de ustedes reciba la mejor educación posible, porque es a través de la educación y la tecnología que vamos a abrir puertas. Es como, en última instancia, llegaremos al cielo,” compartió Rodriguez durante la ceremonia de clausura en el Laboratorio Tecnológico del LATU en Montevideo.
El poder de la educación y la innovación
Juntos, no solo estamos enseñando a los estudiantes cómo usar la tecnología, sino también empoderándolos para construir un futuro donde la innovación sea inclusiva, accesible y sostenible. Estamos emocionados de ver lo que harán los participantes de este año y esperamos con ansias Git Commit 2025.
Sigamos alcanzando el cielo, juntos. 🚀
We’re thrilled to celebrate the third year of Git Commit, our annual program dedicated to bridging the tech skills gap and empowering students across the country. This year, we’re also proud to launch Git Commit 101, our first free online AI course in Spanish, available to students and educators year-round.
Why Uruguay
With one of the highest rates of internet connectivity in Latin America (over 70%), Uruguay leads the region in digital transformation and education. Its commitment to fostering innovation is reflected in its top-five ranking in the Global Innovation Index among Latin American economies, excelling in knowledge outputs and human capital development.
Programs like Git Commit thrive in this environment, aligning perfectly with the nation’s mission to build a knowledge-based economy and create opportunities for underserved communities.
Introducing Git Commit 101
This year, in partnership with Microsoft AI for Good, we launched Git commit 101, a free online AI course streamed live from Microsoft’s AI Co-Innovation Lab in Montevideo—the only lab of its kind in the Southern Hemisphere. The course introduces students to AI fundamentals, GitHub tools, and GitHub Copilot, equipping them with skills they can apply immediately. The online course has already attracted more than 20,000 viewers in its first few weeks.
Our in-person Git Commit event was held at Centros Educativos Impulso and ORT University. Together we hosted 125 students, many from vulnerable backgrounds, from 13 participating institutions. Students heard from inspiring speakers including Dr. Juan M. Lavista Ferres, Corporate Vice President and Chief Data Scientist of the AI for Good Lab at Microsoft, Alejandro Pazos, Microsoft’s Country Manager in Uruguay, Eliane Elbaum, Manager of the Microsoft AI Co-Innovation Lab, and Mario Rodriguez, GitHub’s Chief Product Officer.
“It’s very important that each of you receive the best education possible, because it’s through education and technology that we’re going to open doors. It’s how we’ll ultimately reach for the sky,” Rodriguez shared during the closing ceremony at the LATU Technology Laboratory in Montevideo.
Closing ceremony at the LATU Uruguayan Laboratory of Technology, where we hosted more than 300 guests.
The power of education and innovation
Together, we’re not only teaching students how to use technology, but also empowering them to build a future where innovation is inclusive, accessible, and sustainable. We’re excited to see what this year’s participants go on to do, and look forward to Git Commit 2025.
]]>Multi-Path TCP: Revolutionizing Connectivity, One Path at a Timehttps://blog.imc.re/RSSBOX/rss/cd5286a9.html2025-01-03T14:00:00.000Z2025-01-03T14:00:00.000Z
The Internet is designed to provide multiple paths between two endpoints. Attempts to exploit multi-path opportunities are almost as old as the Internet, culminating in RFCs documenting some of the challenges. Still, today, virtually all end-to-end communication uses only one available path at a time. Why? It turns out that in multi-path setups, even the smallest differences between paths can harm the connection quality due to packet reordering and other issues. As a result, Internet devices usually use a single path and let the routers handle the path selection.
There is another way. Enter Multi-Path TCP (MPTCP), which exploits the presence of multiple interfaces on a device, such as a mobile phone that has both Wi-Fi and cellular antennas, to achieve multi-path connectivity.
MPTCP has had a long history — see the Wikipedia article and the spec (RFC 8684) for details. It's a major extension to the TCP protocol, and historically most of the TCP changes failed to gain traction. However, MPTCP is supposed to be mostly an operating system feature, making it easy to enable. Applications should only need minor code changes to support it.
There is a caveat, however: MPTCP is still fairly immature, and while it can use multiple paths, giving it superpowers over regular TCP, it's not always strictly better than it. Whether MPTCP should be used over TCP is really a case-by-case basis.
In this blog post we show how to set up MPTCP to find out.
Subflows
Internally, MPTCP extends TCP by introducing "subflows". When everything is working, a single TCP connection can be backed by multiple MPTCP subflows, each using different paths. This is a big deal - a single TCP byte stream is now no longer identified by a single 5-tuple. On Linux you can see the subflows with ss -M, like:
Here you can see a single MPTCP connection, composed of two underlying TCP flows.
MPTCP aspirations
Being able to separate the lifetime of a connection from the lifetime of a flow allows MPTCP to address two problems present in classical TCP: aggregation and mobility.
Aggregation: MPTCP can aggregate the bandwidth of many network interfaces. For example, in a data center scenario, it's common to use interface bonding. A single flow can make use of just one physical interface. MPTCP, by being able to launch many subflows, can expose greater overall bandwidth. I'm personally not convinced if this is a real problem. As we'll learn below, modern Linux has a BLESS-like MPTCP scheduler and macOS stack has the "aggregation" mode, so aggregation should work, but I'm not sure how practical it is. However, there are certainly projects that are trying to do link aggregation using MPTCP.
Mobility: On a customer device, a TCP stream is typically broken if the underlying network interface goes away. This is not an uncommon occurrence — consider a smartphone dropping from Wi-Fi to cellular. MPTCP can fix this — it can create and destroy many subflows over the lifetime of a single connection and survive multiple network changes.
Improving reliability for mobile clients is a big deal. While some software can use QUIC, which also works on Multipath Extensions, a large number of classical services still use TCP. A great example is SSH: it would be very nice if you could walk around with a laptop and keep an SSH session open and switch Wi-Fi networks seamlessly, without breaking the connection.
MPTCP work was initially driven by UCLouvain in Belgium. The first serious adoption was on the iPhone. Apparently, users have a tendency to use Siri while they are walking out of their home. It's very common to lose Wi-Fi connectivity while they are doing this. (source)
Implementations
Currently, there are only two major MPTCP implementations — Linux kernel support from v5.6, but realistically you need at least kernel v6.1 (MPTCP is not supported on Android yet) and iOS from version 7 / Mac OS X from 10.10.
Typically, Linux is used on the server side, and iOS/macOS as the client. It's possible to get Linux to work as a client-side, but it's not straightforward, as we'll learn soon. Beware — there is plenty of outdated Linux MPTCP documentation. The code has had a bumpy history and at least two different APIs were proposed. See the Linux kernel source for the mainline API and the mptcp.dev website.
Linux as a server
Conceptually, the MPTCP design is pretty sensible. After the initial TCP handshake, each peer may announce additional addresses (and ports) on which it can be reached. There are two ways of doing this. First, in the handshake TCP packet each peer specifies the "Do not attempt to establish new subflows to this address and port" bit, also known as bit [C], in the MPTCP TCP extensions header.
With this bit cleared, the other peer is free to assume the two-tuple is fine to be reconnected to. Typically, the server allows the client to reuse the server IP/port address. Usually, the client is not listening and disallows the server to connect back to it. There are caveats though. For example, in the context of Cloudflare, where our servers are using Anycast addressing, reconnecting to the server IP/port won't work. Going twice to the IP/port pair is unlikely to reach the same server. For us it makes sense to set this flag, disallowing clients from reconnecting to our server addresses. This can be done on Linux with:
# Linux server sysctl - useful for ECMP or Anycast servers$ sysctl -w net.mptcp.allow_join_initial_addr_port=0
There is also a second way to advertise a listening IP/port. During the lifetime of a connection, a peer can send an ADD-ADDR MPTCP signal which advertises a listening IP/port. This can be managed on Linux by ip mptcp endpoint ... signal, like:
# Linux server - extra listening address$ ip mptcp endpoint add 192.51.100.1 dev eth0 port 4321 signal
With such a config, a Linux peer (typically server) will report the additional IP/port with ADD-ADDR MPTCP signal in an ACK packet, like this:
It's important to realize that either peer can send ADD-ADDR messages. Unusual as it might sound, it's totally fine for the client to advertise extra listening addresses. The most common scenario though, consists of either nobody, or just a server, sending ADD-ADDR.
Technically, to launch an MPTCP socket on Linux, you just need to replace IPPROTO_TCP with IPPROTO_MPTCP in the application code:
In practice, though, this introduces some changes to the sockets API. Currently not all setsockopt's work yet — like TCP_USER_TIMEOUT. Additionally, at this stage, MPTCP is incompatible with kTLS.
Path manager / scheduler
Once the peers have exchanged the address information, MPTCP is ready to kick in and perform the magic. There are two independent pieces of logic that MPTCP handles. First, given the address information, MPTCP must figure out if it should establish additional subflows. The component that decides on this is called "Path Manager". Then, another component called "scheduler" is responsible for choosing a specific subflow to transmit the data over.
Both peers have a path manager, but typically only the client uses it. A path manager has a hard task to launch enough subflows to get the benefits, but not too many subflows which could waste resources. This is where the MPTCP stacks get complicated.
Linux as client
On Linux, path manager is an operating system feature, not an application feature. The in-kernel path manager requires some configuration — it must know which IP addresses and interfaces are okay to start new subflows. This is configured with ip mptcp endpoint ... subflow, like:
$ ip mptcp endpoint add dev wlp1s0 192.0.2.3 subflow # Linux client
This informs the path manager that we (typically a client) own a 192.0.2.3 IP address on interface wlp1s0, and that it's fine to use it as source of a new subflow. There are two additional flags that can be passed here: "backup" and "fullmesh". Maintaining these ip mptcp endpoints on a client is annoying. They need to be added and removed every time networks change. Fortunately, NetworkManager from 1.40 supports managing these by default. If you want to customize the "backup" or "fullmesh" flags, you can do this here (see the documentation):
ubuntu$ cat /etc/NetworkManager/conf.d/95-mptcp.conf# set "subflow" on all managed "ip mptcp endpoints". 0x22 is the default.[connection]connection.mptcp-flags=0x22
Path manager also takes a "limit" setting, to set a cap of additional subflows per MPTCP connection, and limit the received ADD-ADDR messages, like:
$ ip mptcp limits set subflow 4 add_addr_accepted 2 # Linux client
I experimented with the "mobility" use case on my Ubuntu 22 Linux laptop. I repeatedly enabled and disabled Wi-Fi and Ethernet. On new kernels (v6.12), it works, and I was able to hold a reliable MPTCP connection over many interface changes. I was less lucky with the Ubuntu v6.8 kernel. Unfortunately, the default path manager on Linux client only works when the flag "Do not attempt to establish new subflows to this address and port" is cleared on the server. Server-announced ADD-ADDR don't result in new subflows created, unless ip mptcp endpoint has a fullmesh flag.
It feels like the underlying MPTCP transport code works, but the path manager requires a bit more intelligence. With a new kernel, it's possible to get the "interactive" case working out of the box, but not for the ADD-ADDR case.
Custom path manager
Linux allows for two implementations of a path manager component. It can either use built-in kernel implementation (default), or userspace netlink daemon.
$ sysctl -w net.mptcp.pm_type=1 # use userspace path manager
However, from what I found there is no serious implementation of configurable userspace path manager. The existing implementations don't do much, and the API seemsimmature yet.
Scheduler and BPF extensions
Thus far we've covered Path Manager, but what about the scheduler that chooses which link to actually use? It seems that on Linux there is only one built-in "default" scheduler, and it can do basic failover on packet loss. The developers want to write MPTCP schedulers in BPF, and this work is in-progress.
macOS
As opposed to Linux, macOS and iOS expose a raw MPTCP API. On those operating systems, path manager is not handled by the kernel, but instead can be an application responsibility. The exposed low-level API is based on connectx(). For example, here's an example of obscure code that establishes one connection with two subflows:
int sock = socket(AF_MULTIPATH, SOCK_STREAM, 0);connectx(sock, ..., &cid1);connectx(sock, ..., &cid2);
This powerful API is hard to use though, as it would require every application to listen for network changes. Fortunately, macOS and iOS also expose higher-level APIs. One example is nw_connection in C, which uses nw_parameters_set_multipath_service.
Another, more common example is using Network.framework, and would look like this:
Handover Mode: Tries to minimize cellular. Uses only Wi-Fi. Uses cellular only when Wi-Fi Assist is enabled and makes such a decision.
Interactive Mode: Used for Siri. Reduces latency. Only for low-bandwidth flows.
Aggregation Mode: Enables resource pooling but it's only available for developer accounts and not deployable.
The MPTCP API is nicely integrated with the iPhone "Wi-Fi Assist" feature. While the official documentation is lacking, it's possible to find sources explaining how it actually works. I was able to successfully test both the cleared "Do not attempt to establish new subflows" bit and ADD-ADDR scenarios. Hurray!
IPv6 caveat
Sadly, MPTCP IPv6 has a caveat. Since IPv6 addresses are long, and MPTCP uses the space-constrained TCP Extensions field, there is not enough room for ADD-ADDR messages if TCP timestamps are enabled. If you want to use MPTCP and IPv6, it's something to consider.
Summary
I find MPTCP very exciting, being one of a few deployable serious TCP extensions. However, current implementations are limited. My experimentation showed that the only practical scenario where currently MPTCP might be useful is:
Linux as a server
macOS/iOS as a client
"interactive" use case
With a bit of effort, Linux can be made to work as a client.
Don't get me wrong, Linux developers did tremendous work to get where we are, but, in my opinion for any serious out-of-the-box use case, we're not there yet. I'm optimistic that Linux can develop a good MPTCP client story relatively soon, and the possibility of implementing the Path manager and Scheduler in BPF is really enticing.
Time will tell if MPTCP succeeds — it's been 15 years in the making. In the meantime, Multi-Path QUIC is under active development, but it's even further from being usable at this stage.
We're not quite sure if it makes sense for Cloudflare to support MPTCP. Reach out if you have a use case in mind!
Shoutout to Matthieu Baerts for tremendous help with this blog post.
]]>Behind the Scenes With Stream Live, Cloudflare’s Live Streaming Servicehttps://blog.imc.re/RSSBOX/rss/101c747.html2025-01-02T14:00:00.000Z2025-01-02T14:00:00.000Z
Cloudflare announced Stream Live for open beta in 2021, and in 2022 we went GA. While we talked about the experience of using it and the value it delivers to customers, we didn’t talk about how we built it. So let’s talk about Stream Live’s design, and how it leverages the distributed nature of Cloudflare’s network, rather than centralized locations as many other live services do. Ultimately, our goals are to keep our content ingest as close to broadcasters as possible, our content delivery as close to viewers as possible, and to retain our ability to handle unexpected use cases.
At a high level, Stream Live accepts audio/video content from broadcasters and makes that content available to viewers around the world in real time through the Cloudflare network, which reaches more than 330 cities in over 120 countries. Hence, there are two sides to this: ingesting data from broadcasters and delivering encoded content to viewers. Both sides are built on a combination of internal systems and Cloudflare products, including Cloudflare Workers, Durable Objects, Spectrum, and, of course, Cache.
Let’s start on the ingest side.
Ingesting a broadcast
Broadcasters generate content in real time, as a series of video and audio frames, and it needs to be transmitted to Cloudflare over the Internet. To communicate with us, they choose a protocol such as RTMPS (Real Time Messaging Protocol Secure), SRT (Secure Reliable Transport), or WHIP (WebRTC-HTTP ingestion protocol) that defines how their content is packaged and transmitted. Each of these protocols is a way to transmit audio and video frames with various tradeoffs.
Regardless of the chosen protocol, the ingest lifecycle looks fairly similar. Broadcasters connect to an Anycast IP address using either a custom ingest domain or our default live.cloudflare.com. Both options direct to an IP address advertised from every Cloudflare data center around the world, minimizing the latency for broadcasters (both big and small) to our ingest points.
When the media content arrives at the Cloudflare server servicing the connection, it is first handled by a Spectrum application. Spectrum saved us time by implementing TLS termination for RTMPS, blocking potential DDOS attacks, and a few other protocol-specific benefits, such as our ability to support SRT broadcasters whose clients don't support the Stream ID portion of the SRT protocol. Those broadcasters assume their connection can be fully identified by connecting to a specific port, which was a challenge for our multitenant service. We use Spectrum to expose many listening ports to specific broadcasters which get wrapped up in Simple Proxy Protocol and sent to one ingest service port. This is important, as our SRT implementation spends a non-trivial amount of CPU for each listening port, whereas Spectrum spends effectively nothing. In any case, Spectrum forwards all connections to the ingest service running on the same server.
Our ingest service handles receiving content from broadcasters, forwarding to live outputs, and recording for HLS/DASH viewing. The ingest service is written in Go and acts on configuration and broadcast state stored in our Live Config Durable Objects. One Durable Object instance represents one live input; it contains both the customer configuration and the ongoing state of each broadcast.
We chose Durable Objects over a centralized database since they are not coupled to any specific data center, allowing them to be closer to each of our geographically distributed broadcasters while remaining highly available. In terms of customer configuration, we use Durable Objects to store everything defined when creating or modifying the live input.
We store the canonical state of the broadcast in the Durable Object. That includes timestamp metadata for keeping times monotonic, connection status, and which ingest service instance owns the broadcast. Any content received by the ingest service needs to be acknowledged and indexed by the Durable Object before it is made available to viewers. This splits our state into two types, ‘volatile’ and ‘committed’. Volatile state is content the ingest has received but not yet told the Durable Object about. Committed state is content the Durable Object has acknowledged and is used as a sync point for any other ingest service instance in the event they claim ownership of the broadcast.
This split between volatile and committed states is how we support broadcasters resuming a live broadcast after disconnecting and reconnecting for whatever reason, including a broadcaster network blip. That’s normally a relatively easy problem when you have centralized ingestion and state storage. Since broadcasters connect to an arbitrary server due to Anycast, we needed to get more creative in making sure that whichever server receives the reconnect has the data to continue the broadcast.
The ingest service itself is written as a relay, taking packets from one input stream and mapping them to multiple output streams. At the top level, the relay is implemented as two coupled ‘for’ loops that communicate over a Go channel, one for sending and one for receiving. When we receive data, we internally normalize it depending on the protocol. For example, some protocols send video packets as delimiter-based (annex B h264 NALU), but other protocols or file formats expect length-prefixed packets (avcc h264 NALU).
A special case of relay outputs is our Live Playback and Recording functionality. When this is enabled, we copy and sanitize the incoming packets. Specifically, we issue monotonically increasing timestamps, which solves many issues we’ve encountered with various customer encoders. To ensure everything is aligned, we drop severely misaligned audio/video blocks, something typically seen at the start of broadcasts. Those sanitized packets are packaged into fragmented MP4s on keyframe boundaries. We call those fragmented MP4s ‘original segments.’ Our ingest service stores the original segments in R2 and lets the live config Durable Object know the segment exists and how long it is.
These original segments are considered the canonical copy of the content a customer has uploaded to us, and are reused when transitioning from live to on-demand. Since these are required to serve live viewers, this is why we don’t currently provide an option to decouple live playback from recording. Supporting live playback automatically implements on-demand recording, with some state management overhead.
Viewing the broadcast
At this point, we’ve ingested the content, cleaned it up, and durably stored it. Lets talk about how viewers watch this content.
Most online video is delivered in short 'segments’, multi-second chunks of content. Splitting the output allows content to be progressively emitted and is very cache-friendly. The protocols viewers use to request that content are typically HLS (HTTP Live Streaming) or MPEG-DASH (Dynamic Adaptive Streaming over HTTP). WHEP (WebRTC-HTTP Egress Protocol) is a non-segmented viewing method available for real-time viewing in some cases. However, we’ll focus here on playback using segmented content with HLS or DASH, since that’s most of Stream Live’s usage today.
To start viewing, a video player will request an HLS or DASH playlist which describes the attributes of the media content and acts as an index for each segment. The playlist tells us which segments map to what point of the video’s timeline. Those segments are inserted into a playback buffer to be decoded and displayed by a client’s player. Examples here will use HLS, which is a newline delimited format. The typical alternative is DASH, which is XML based.
First, ‘primary’ playlists describe which renditions of the same content are available, as well as where to get their specific index files. Those renditions can vary in bitrate, codec, framerate, resolution, etc. If you’ve ever picked ‘1080p’ from a video player quality menu, the player knew those qualities were available using this or similar methods. When selecting quality automatically, players choose the best rendition for the viewing machine’s capabilities (such as the ability to decode a certain codec) and network constraints. We use an internal representation of tracks (what kind of content, i.e. video), renditions (content parameters, i.e. 1080p), and muxings (where to find that content, i.e. in R2 bucket N or OTFE with call M) to generate both HLS and DASH, as the two formats contain nearly the same data, except organized differently. Here’s a simplified example of a ‘multi-variant’ or ‘primary’ HLS playlist that Stream generated. It includes some annotations to explain the components.
#EXTM3U#EXT-X-VERSION:6#EXT-X-INDEPENDENT-SEGMENTS#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="group_audio",NAME="original",LANGUAGE="en-0a76e0ad",DEFAULT=YES,AUTOSELECT=YES,URI="stream_2.m3u8" <- Audio track description + URL path...#EXT-X-STREAM-INF:RESOLUTION=426x240,CODECS="avc1.42c015,mp4a.40.2",BANDWIDTH=149084,AVERAGE-BANDWIDTH=145135,SCORE=1.0,FRAME-RATE=30.000,AUDIO="group_audio" <- description of variant contentsstream_1.m3u8 <- URL path to fetch variant
Second, ‘variant’ or ‘stream’ playlists contain a list of segments that can be downloaded and viewed for each rendition. These are used for both live and on-demand viewing. The difference is on-demand playlists contain a flag indicating no more content will be added to the index whereas live omits that flag to allow it to append content in future requests. As a result, you’ll see video players download M3U8 (HLS) or MPD (DASH) files approximately every 1–10 seconds when viewing a broadcast, looking for updated content.
#EXTM3U#EXT-X-VERSION:6#EXT-X-MEDIA-SEQUENCE:89281 <- Indicates position of sliding window#EXT-X-TARGETDURATION:6#EXT-X-MAP:URI=".../init.mp4" <- Initialization data#EXTINF:4.000, <- How long the segment.../seg_89281.mp4 <- Location of a segment#EXTINF:4.000,.../seg_89282.mp4#EXTINF:4.000,.../seg_89283.mp4
To serve all requests from viewers, we use a Cloudflare Worker called ‘delivery-worker’. It handles all requests for Stream content, and had its first commit in 2017, making it one of the earliest production-facing workers at Cloudflare. Since it's a worker that executes as close to a viewer as possible, it frees us up to spend more time on content and metadata performance rather than where our logic runs. It delivers content, renders playlists, and performs a variety of other functions. For rendering playlists, the worker transforms the broadcast state from the durable object, as mentioned earlier.
When clients ask for the encoded media content the playlist advertised to them, delivery-worker will send a subrequest to our OTFE (on-the-fly-encoder) service that transits the Cloudflare network. That request describes the format of content requested, i.e. the video stream of segment 89282 at a 1280x720 resolution encoded using AVC (aka H.264) capped CRF with some specified bitrate cap. Then, OTFE will encode the original segment to output the specified configuration.
On-the-fly encoding is more efficient than always-encoding, which is the approach most other platforms take. If there are no viewers watching a specific quality level, or watching the broadcast at all, then we aren’t encoding it. That saves power, CPU time, RAM, network, and cache space. Doing nothing is always more efficient than doing something. This applies to a variety of customer use cases, since not all broadcasts have many viewers widely distributed across a range of connection qualities. For one case, consider when serving live broadcasts that are primarily viewed on high quality connections — encoding 240p or 360p variants would probably go to waste most of the time. For a more extreme case, there are situations where you definitely want recording enabled for live content, but viewing is an exceptional situation, such as for security cameras or dashcams. Of course, we have many customers that have active viewership for their broadcasts and this architecture allows us to serve both use cases.
On-the-fly encoding has a tradeoff: it is hard to implement for “media-correctness” and performance reasons. Media-correctness is important to ensuring smooth playback; individual segments need to have exactly the right start time and duration, or you get stuttering playback, audio/video desync, or entirely unwatchable content. Perfecting our media output requires fine-tuning our encoding, deep-diving into specs, and adjusting fragmented MP4s — especially since most encoders aren’t designed for per-segment encoding. For performance, we hide encoding delay by aggressively prefetching segments from delivery-worker. When a viewer initiates a request for Segment N, we initiate encoding of segment N+1. Since that logic is implemented as a worker, we can also easily add or iterate the prefetching however we want.
This encoding flow stands on top of the Cloudflare network, which also provides us with tiered caching and request coalescing. Request coalescing is the key to supporting many viewers simultaneously but only encoding once by enforcing that, for any number of viewers requesting the same encoded content, only one of those requests will make it to our encoder origin — thanks to Cache.
That’s how Stream Live works at a high level. We ingest content from users, send it to any desired live outputs, transcode it for viewing, and give viewers a choice of quality levels, with a lot of backend complexity hidden behind a friendly API. All of this is built on top of the Cloudflare network, with Cloudflare as Customer Zero for our own products and services, using the same as the ones available to our customers.
There’s a lot more we can write about for problems we’ve solved in building Stream Live over the last few years, but those will have to be for a future blog post. If this mix of media and distributed system problems discussed here sound interesting, the Cloudflare Media Platform is hiring for several roles. Apply and join us!
]]>The Forecast Is Clear: Clouds on E-Paper, Powered by the Cloudhttps://blog.imc.re/RSSBOX/rss/90648e4f.html2024-12-31T14:00:00.000Z2024-12-31T14:00:00.000Z
I’ve noticed that many shops are increasingly using e-paper displays. They’re impressive: high contrast, no backlight, and no visible cables. Unlike most electronics, these displays are seamlessly integrated and feel very natural. This got me wondering: is it possible to use such a display for a pet project? I want to experiment with this technology myself.
My main goal in this project is to understand the hardware and its capabilities. Here, I'll be using an e-paper display to show the current weather, but at its core, I’m simply feeding data from a website to the display. While it sounds straightforward, it actually requires three layers of software to pull off. Still, it’s a fun challenge and a great opportunity to work with both embedded hardware and Cloudflare Workers.
Sourcing the hardware
For this project, I'm using components from Waveshare. They offer a variety of e-paper displays, ranging from credit card-sized to A4-sized models. I chose the 7.5-inch, two-color "e-Paper (G)" display. For the controller, I'm using a Waveshare ESP32-based universal board. With just these two components — a display and a controller — I was ready to get started.
When the components arrived, I carefully connected the display’s ribbon cable to the ESP32 board. Even though this step isn’t documented anywhere, it was simple and almost impossible to get wrong. Best of all, no soldering was needed!
That’s pretty much it for the hardware setup! I’m keeping the device powered with a 5V supply through a micro-USB connection.
This was my first time working with the ESP32 CPU family, and I’m really impressed. It’s a system-on-chip controller with built-in Bluetooth and Wi-Fi. It’s relatively fast, very power-efficient, and quite popular in DSP (digital signal processing) applications. For example, your audio device might be powered by a CPU like this. Interestingly, the newer models have switched to the RISC-V instruction set.
For our purposes, we’ll only scratch the surface of what the ESP32 is capable of. The chip is straightforward to work with, thanks to the familiar Arduino environment. A great starting point is the demo provided by Waveshare. It sets up a web page where you can easily upload a custom image to the display.
Install "Additional Boards Manager URL" as per the instructions, and install the "esp32 by expressif" bundle.
Open the "Loader_esp32wf" example downloaded from waveshare.
Change the Wi-Fi name, password and IP address in the Arduino IDE srvr.h tab.
Once everything is set up, you should be able to connect to the ESP32’s IP address and use the simple web interface to upload an image to the display.
With a simple click of the "Upload Image" button, the magic happens: the e-paper display comes to life, showcasing the uploaded image.
With the demo up and running, we can move on to the next step: figuring out how to render a web page on the e-paper display.
Three layers of software
The ESP32 comes with some limitations. It has 520 KiB of RAM, 4 MiB of flash, and a 240 MHz clock speed. While this is fine for tasks like connecting to Wi-Fi or fetching a simple URL, it’s not powerful enough for more demanding tasks, such as parsing JSON or rendering an entire web page.
There are basic Arduino libraries for handling bitmaps, which can draw rectangles and render simple fonts, but manually managing layout doesn't sound appealing to me. A better approach is to play to the ESP32’s strengths — fetching and displaying bitmaps — and delegate the more complex task of HTML rendering to a more powerful server.
Let’s break the problem into three layers:
ESP32 (Display Layer): The ESP32 will periodically, say every minute, fetch a pre-rendered bitmap from the server and display it on the e-paper screen. This keeps the ESP32's tasks lightweight and manageable.
Server A (Rendering Layer): This server will fetch the desired website, render it, and rasterize it into a bitmap format. Its job is to prepare a bitmap that the ESP32 can handle without additional processing.
Server B (Content Layer): This server hosts the actual website with the HTML and CSS content. In this case, it will provide the local weather data in a styled format, ready to be fetched and rendered by Server A.
ESP32 (Display Layer)
The ESP32 provides some great higher-level libraries to simplify development. For this project, we’ll need three key components:
Then, it fetches a rendered bitmap from an HTTP endpoint
Then it pushes it to the e-paper display if needed
Waits a minute
And repeats the whole process forever
E-paper displays typically start to degrade after about one million refresh cycles. To preserve the display's lifespan, I’m being extra careful to avoid unnecessary refreshes.
Server A (Rendering Layer)
Now for the exciting part! We need an online service that can fetch a website, render it, rasterize it to fit our small monochromatic display, and return it as a display-sized binary blob. Initially, I considered using headless Chrome paired with an ImageMagick script, but then I discovered Cloudflare’s Browser Rendering API, which fits our needs perfectly.
This API can be used quite trivially and nicely fits our needs. Here's the typescript worker code, and there are two particularly interesting parts: handling a remote browser and dithering.
Remote Browser API
First, see how easy it is to render a website as a PNG using Browser Rendering:
await page.goto(url); img = (await page.screenshot()) as Buffer;
I’m genuinely surprised at how practical and effective this approach is. While the remote browser startup isn’t exactly fast — it can take a few seconds to generate the screenshot — it’s not an issue for my use case. The delay is perfectly acceptable, especially considering how much work is offloaded to the cloud.
Dithering
To prepare the bitmap for the ESP32, we need to decode the PNG, reduce the color palette to monochromatic, and apply dithering. Here's the dithering code:
function ditherTwoBits(px: Buffer, width: number, height: number ): Buffer { px = new Float32Array(px);
This was my first time experimenting with dithering, and it’s been a lot of fun! I was surprised by how straightforward the process is and that it’s fully deterministic. Now that I understand the details of the algorithm, I can’t help but notice its subtle side effects everywhere — in printed materials, on screens, and even in design choices around me. It’s fascinating how something so simple has such a broad impact!
To deploy this code as a Cloudflare Worker, you only need to install the required dependencies, configure the wrangler.toml file, and publish the code. Here’s a step-by-step guide:
2025-01-e-paper/worker-render-raster$ npx wrangler dev --remote
⛅️ wrangler 3.99.0
Your worker has access to the following bindings:
KV Namespaces:
KV: XXX
Browser:
Name: BROWSER [wrangler:inf] Ready on http://localhost:46131 ⎔ Starting remote preview… Total Upload: 755.39 KiB / gzip: 149.05 KiB ╭─────────────────────────────────────────────────────────────────────────────────────────────────╮ │ [b] open a browser, [d] open devtools, [l] turn on local mode, [c] clear console, [x] to exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────╯
With everything set up, you can now open a browser and see a rendered and rasterized version of a website, processed through your Cloudflare Worker! For example, here’s how the 1.1.1.1 page looks in a 800x480 monochromatic resolution, complete with dithering:
This demonstrates how effectively the Worker can handle rendering, rasterizing, and adapting web content for an e-paper display. It’s quite satisfying to see the pipeline in action.
Server B (Content Layer)
To create the weather panel, I designed a simple HTML and CSS page and published it as another Cloudflare Worker. This time, I used Python in Cloudflare Workers because it felt more straightforward, especially since the site needs to query an external weather API. The simplicity of the code was surprising and made the process smooth.
Here’s how it appears in a normal browser compared to the rendered and rasterized version by our worker:
Summary
Finally, the display deserves a proper frame. Here’s the finished version:
I started this project wanting to experiment with an e-paper display hardware, but I ended up spending most of my time writing software—and it turned out to be surprisingly enjoyable across all layers:
ESP32: The CPU is fantastic. Programming it is straightforward, thanks to powerful built-in libraries that simplify development.
Cloudflare Worker Browser Rendering: This is an underrated but incredibly powerful technology. It made implementing features like the Floyd–Steinberg dithering algorithm surprisingly easy.
Cloudflare Worker Python: Although still in beta, it worked flawlessly for my needs and was a great fit for handling API requests and serving dynamic content.
It’s remarkable how much you can achieve with relatively inexpensive hardware and free Cloudflare services.
]]>Open Sourcing H3i: A Command Line Tool and Library for Low-Level HTTP/3 Testing and Debugginghttps://blog.imc.re/RSSBOX/rss/5a651ac4.html2024-12-30T14:00:00.000Z2024-12-30T14:00:00.000Z
Have you ever built a piece of IKEA furniture, or put together a LEGO set, by following the instructions closely and only at the end realized at some point you didn't quite follow them correctly? The final result might be close to what was intended, but there's a nagging thought that maybe, just maybe, it's not as rock steady or functional as it could have been.
Internet protocol specifications are instructions designed for engineers to build things. Protocol designers take great care to ensure the documents they produce are clear. The standardization process gathers consensus and review from experts in the field, to further ensure document quality. Any reasonably skilled engineer should be able to take a specification and produce a performant, reliable, and secure implementation. The Internet is central to everyone's lives, and we depend on these implementations. Any deviations from the specification can put us at risk. For example, mishandling of malformed requests can allow attacks such as request smuggling.
h3i is a binary command line tool and Rust library designed for low-level testing and debugging of HTTP/3, which runs over QUIC. h3i is free and open source as part of Cloudflare's quiche project. In this post we'll explain the motivation behind developing h3i, how we use it to help develop robust and safe standards-compliant software and production systems, and how you can similarly use it to test your own software or services. If you just want to jump into how to use h3i, go to the h3i command line tool section.
A recap of QUIC and HTTP/3
QUIC is a secure-by-default transport protocol that provides performance advantages compared to TCP and TLS via a more efficient handshake, along with stream multiplexing that provides head-of-line blocking avoidance. HTTP/3 is an application protocol that maps HTTP semantics to QUIC, such as defining how HTTP requests and responses are assigned to individual QUIC streams.
Cloudflare has supported QUIC on our global network in some shape or form since 2018. We started while the Internet Engineering Task Force (IETF) was earnestly standardizing the protocol, working through early iterations and using interoperability testing and experience to help provide feedback for the standards process. We launched support for QUIC version 1 and HTTP/3 as soon as RFC 9000 (and its accompanying specifications) were published in 2021.
We work on the Protocols team, who own the ingress proxy into the Cloudflare network. This is essentially Cloudflare’s “front door” — HTTP requests that come to Cloudflare from the Internet pass through us first. The majority of requests are passed onwards to things like rulesets, workers, caches, or a customer origin. However, you might be surprised that many requests don't ever make it that far because they are, in some way, invalid or malformed. Servers listening on the Internet have to be robust to traffic that is not RFC compliant, whether caused by accident or malicious intent.
The Protocols team actively participates in IETF standardization work and has also helped build and maintain other Cloudflare services that leverage quiche for QUIC and HTTP/3, from the proxies that help iCloud Private Relay via MASQUE proxying, to replacing WARP's use of Wireguard with MASQUE, and beyond.
Throughout all of these different use cases, it is important for us to extensively test all aspects of the protocols. A deep dive into protocol details is a blog post (or three) in its own right. So let's take a thin slice across HTTP to help illustrate the concepts.
HTTP Semantics are common to all versions of HTTP — the overall architecture, terminology, and protocol aspects such as request and response messages, methods, status codes, header and trailer fields, message content, and much more. Each individual HTTP version defines how semantics are transformed into a "wire format" for exchange over the Internet. You can read more about HTTP/1.1 and HTTP/2 in some of our previous blogposts.
With HTTP/3, HTTP request and response messages are split into a series of binary frames. HEADERS frames carry a representation of HTTP metadata (method, path, status code, field lines). The payload of the frame is the encoded QPACK compression output. DATA frames carry HTTP content (aka "message body"). In order to exchange these frames, HTTP/3 relies on QUIC streams. These provide an ordered and reliable byte stream and each have an identifier (ID) that is unique within the scope of a connection. There are four different stream types, denominated by the two least significant bits of the ID.
As a simple example, assuming a QUIC connection has already been established, a client can make a GET request and receive a 200 OK response with an HTML body using the follow sequence:
Client allocates the first available client-initiated bidirectional QUIC stream. (The IDs start at 0, then 4, 8, 12 and so on)
Client sends the request HEADERS frame on the stream and sets the stream's FIN bit to mark the end of stream.
Server receives the request HEADERS frame and validates it against RFC 9114 rules. If accepted, it processes the request and prepares the response.
Server sends the response HEADERS frame on the same stream.
Server sends the response DATA frame on the same stream and sets the FIN bit.
Client receives the response frames and validates them. If accepted, the content is presented to the user.
At the QUIC layer, stream data is split into STREAM frames, which are sent in QUIC packets over UDP. QUIC deals with any loss detection and recovery, helping to ensure stream data is reliable. The layer cake diagram below provides a handy comparison of how HTTP/1.1, HTTP/2 and HTTP/3 use TCP, UDP and IP.
Background on testing QUIC and HTTP/3 at Cloudflare
The Protocols team has a diverse set of automated test tools that exercise our ingress proxy software in order to ensure it can stand up to the deluge that the Internet can throw at it. Just like a bouncer at a nightclub front door, we need to prevent as much bad traffic as possible before it gets inside and potentially causes damage.
HTTP/2 and HTTP/3 share several concepts. When we started developing early HTTP/3 support, we'd already learned a lot from production experience with HTTP/2. While HTTP/2 addressed many issues with HTTP/1.1 (especially problems like request smuggling, caused by its ASCII-based message delineation), HTTP/2 also added complexity and new avenues for attack. Security is an ongoing process, and the Protocols team continually hardens our software and systems to threats. For example, mitigating the range of denial-of-service attacks identified by Netflix in 2019, or the HTTP/2 Rapid Reset attacks of 2023.
For testing HTTP/2, we rely on the Python Requests library for testing conventional HTTP exchanges. However, that mostly only exercises HEADERS and DATA frames. There are eight other frame types and a plethora of ways that they can interact (hence the new attack vectors mentioned above). In order to get full testing coverage, we have to break down into the lower layer h2 library, which allows exact frame-by-frame control. However, even that is not always enough. Libraries tend to want to follow the RFC rules and prevent their users from doing "the wrong thing". This is entirely logical for most purposes. For our needs though, we need to take off the safety guards just like any potential attackers might do. We have a few cases where the best way to exercise certain traffic patterns is to handcraft HTTP/2 frames in a hex editor, store that as binary, and replay it with a tool such as OpenSSL s_client.
We knew we'd need similar testing approaches for HTTP/3. However, when we started in 2018, there weren't many other suitable client implementations. The rate of iteration on the specifications also meant it was hard to always keep in sync. So we built tests on quiche, using a mix of our quiche-client and http3_test. Over time, the python library aioquic has matured, and we have used it to add a range of lower-layer tests that break or bend HTTP/3 rules, in order to prove our proxies are robust.
Finally, we would be remiss not to mention that all the tests in our ingress proxy are in addition to the suite of over 500 integration tests that run on the quiche project itself.
Making HTTP/3 testing more accessible and maintainable with h3i
While we are happy with the coverage of our current tests, the smorgasbord of test tools makes it hard to know what to reach for when adding new tests. For example, we've had cases where aioquic's safety guards prevent us from doing something, and it has needed a patch or workaround. This sort of thing requires a time investment just to debug/develop the tests.
We believe it shouldn't take a protocol or code expert to develop what are often very simple to describe tests. While it is important to provide guide rails for the majority of conventional use cases, it is also important to provide accessible methods for taking them off.
Let's consider a simple example. In HTTP/3 there is something called the control stream. It's used to exchange frames such as SETTINGS, which affect the HTTP/3 connection. RFC 9114 Section 6.2.1 states:
Each side MUST initiate a single control stream at the beginning of the connection and send its SETTINGS frame as the first frame on this stream. If the first frame of the control stream is any other frame type, this MUST be treated as a connection error of type H3_MISSING_SETTINGS. Only one control stream per peer is permitted; receipt of a second stream claiming to be a control stream MUST be treated as a connection error of type H3_STREAM_CREATION_ERROR. The sender MUST NOT close the control stream, and the receiver MUST NOT request that the sender close the control stream. If either control stream is closed at any point, this MUST be treated as a connection error of type H3_CLOSED_CRITICAL_STREAM. Connection errors are described in Section 8.
There are many tests we can conjure up just from that paragraph:
Send a non-SETTINGS frame as the first frame on the control stream.
Open two control streams.
Open a control stream and then close it with a FIN bit.
Open a control stream and then reset it with a RESET_STREAM QUIC frame.
Wait for the peer to open a control stream and then ask for it to be reset with a STOP_SENDING QUIC frame.
All of the above actions should cause a remote peer that has implemented the RFC properly to close the connection. Therefore, it is not in the interest of the local client or server applications to ever do these actions.
Many QUIC and HTTP/3 implementations are developed as libraries that are integrated into client or server applications. There may be an extensive set of unit or integration tests of the library checking RFC rules. However, it is also important to run the same tests on the integrated assembly of library and application, since it's all too common that an unhandled/mishandled library error can cascade to cause issues in upper layers. For instance, the HTTP/2 Rapid Reset attacks affected Cloudflare due to their impact on how one service spoke to another.
We've developed h3i, a command line tool and library, to make testing more accessible and maintainable for all. We started with a client that can exercise servers, since that's what our focus has been. Future developments could support the opposite, a server that behaves in unusual ways in order to exercise clients.
Note: h3i is not intended to be a production client! Its flexibility may cause issues that are not observed in other production-oriented clients. It is also not intended to be used for any type of performance testing and measurement.
The h3i command line tool
The primary purpose of the h3i command line tool is quick low-level debugging and exploratory testing. Rather than worrying about writing code or a test script, users can quickly run an ad-hoc client test against a target, guided by interactive prompts.
In the simplest case, you can think of h3i a bit like curl but with access to some extra HTTP/3 parameters. In the example below, we issue a request to https://cloudflare-quic.com/ and receive a response.
Walking through a simple GET with h3i step-by-step:
Grab a copy of the h3i binary either by running cargo install h3i or cloning the quiche source repo at https://github.com/cloudflare/quiche/. Both methods assume you have some familiarity with Rust and Cargo. See the cargo documentation for more information.
cargo install will place the binary on your path, so you can then just run it by executing h3i.
If running from source, navigate to the quiche/h3i directory and then use cargo run.
Run the binary and provide the name and port of the target server. If the port is omitted, the default value 443 is assumed. E.g, cargo run cloudflare-quic.com
h3i then enters the action prompting phase. A series of one or more HTTP/3 actions can be queued up, such as sending frames, opening or terminating streams, or waiting on data from the server. The full set of options is documented in the readme.
The prompting interface adapts to keyboard inputs and supports tab completion.
In the example above, the headers action is selected, which walks through populating the fields in a HEADERS frame. It includes mandatory fields from RFC 9114 for convenience. If a test requires omitting these, the headers_no_pseudo can be used instead.
The commit prompt choice finalizes the action list and moves to the connection phase. h3i initiates a QUIC connection to the server identified in step 2. Once connected, actions are executed in order.
By default, h3i reports some limited information about the frames the server sent. To get more detailed information, the RUST_LOG environment can be set with either debug or trace levels.
Instant record and replay, powered by qlog
It can be fun to play around with the h3i command line tool to see how different servers respond to different combinations or sequences of actions. Occasionally, you'll find a certain set that you want to run over and over again, or share with a friend or colleague. Having to manually enter the prompts repeatedly, or share screenshots of the h3i input can turn tedious. Fortunately, h3i records all the actions in a log file by default — the file path is printed immediately after h3i starts. The format of this file is based on qlog, an in-progress standard in development at the IETF for network protocol logging. It’s a perfect fit for our low-level needs.
h3i logs can be replayed using the --qlog-input option. You can change the target server host and port, and keep all the same actions. However, most servers will validate the :authority pseudo-header or Host header contained in a HEADERS frame. The --replay-host-override option allows changing these fields without needing to modify the file by hand.
And yes, qlog files are human-readable text in the JSON-SEQ format. So you can also just write these by hand in the first place if you like! However, if you're going to start writing things, maybe Rust is your preferred option…
Using the h3i library to send a malformed request with Rust
In our previous example, we just sent a valid request so there wasn't anything interesting to observe. Where h3i really shines is in generating traffic that isn't RFC compliant, such as malformed HTTP messages, invalid frame sequences, or other actions on streams. This helps determine if a server is acting robustly and defensively.
Let's explore this more with an example of HTTP content-length mismatch. RFC 9114 section 4.1.2 specifies:
A request or response that is defined as having content when it contains a Content-Length header field (Section 8.6 of [HTTP]) is malformed if the value of the Content-Length header field does not equal the sum of the DATA frame lengths received. A response that is defined as never having content, even when a Content-Length is present, can have a non-zero Content-Length header field even though no content is included in DATA frames.
Intermediaries that process HTTP requests or responses (i.e., any intermediary not acting as a tunnel) MUST NOT forward a malformed request or response. Malformed requests or responses that are detected MUST be treated as a stream error of type H3_MESSAGE_ERROR.
For malformed requests, a server MAY send an HTTP response indicating the error prior to closing or resetting the stream.
There are good reasons that the RFC is so strict about handling mismatched content lengths. They can be a vector for desynchronization attacks (similar to request smuggling), especially when a proxy is converting inbound HTTP/3 to outbound HTTP/1.1.
We've provided an example of how to use the h3i Rust library to write a tailor-made test client that sends a mismatched content length request. It sends a Content-Length header of 5, but its body payload is “test”, which is only 4 bytes. It then waits for the server to respond, after which it explicitly closes the connection by sending a QUIC CONNECTION_CLOSE frame.
When running low-level tests, it can be interesting to also take a packet capture (pcap) and observe what is happening on the wire. Since QUIC is an encrypted transport, we'll need to use the SSLKEYLOG environment variable to capture the session keys so that tools like Wireshark can decrypt and dissect.
To follow along at home, clone a copy of the quiche repository, start a packet capture on the appropriate network interface and then run:
cd quiche/h3iSSLKEYLOGFILE="h3i-example.keys" cargo run --example content_length_mismatch
In our decrypted capture, we see the expected sequence of handshake, request, response, and then closure.
Surveying the example code
The example is a simple binary app with a main() entry point. Let's survey the key elements.
First, we set up an h3i configuration to a target server:
let config = Config::new() .with_host_port("cloudflare-quic.com".to_string()) .with_idle_timeout(2000) .build() .unwrap();
The idle timeout is a QUIC concept which tells each endpoint when it should close the connection if the connection has been idle. This prevents endpoints from spinning idly if the peer hasn’t closed the connection. h3i’s default is 30 seconds, which can be too long for tests, so we set ours to 2 seconds here.
Next, we define a set of request headers and encode them with QPACK compression, ready to put in a HEADERS frame. Note that h3i does provide a send_headers_frame helper method which does this for you, but the example does it manually for clarity:
let headers = vec![ Header::new(b":method", b"POST"), Header::new(b":scheme", b"https"), Header::new(b":authority", b"cloudflare-quic.com"), Header::new(b":path", b"/"), // We say that we're going to send a body with 5 bytes... Header::new(b"content-length", b"5"), ];
let header_block = encode_header_block(&headers).unwrap();</code></pre>
Then, we define the set of h3i actions that we want to execute in order: send HEADERS, send a too-short DATA frame, wait for the server's HEADERS, then close the connection.
let actions = vec![ Action::SendHeadersFrame { stream_id: STREAM_ID, fin_stream: false, headers, frame: Frame::Headers { header_block }, }, Action::SendFrame { stream_id: STREAM_ID, fin_stream: true, frame: Frame::Data { // ...but, in actuality, we only send 4 bytes. This should yield a // 400 Bad Request response from an RFC-compliant // server: https://datatracker.ietf.org/doc/html/rfc9114#section-4.1.2-3 payload: b"test".to_vec(), }, }, Action::Wait { wait_type: WaitType::StreamEvent(StreamEvent { stream_id: STREAM_ID, event_type: StreamEventType::Headers, }), }, Action::ConnectionClose { error: quiche::ConnectionError { is_app: true, error_code: quiche::h3::WireErrorCode::NoError as u64, reason: vec![], }, }, ];
Finally, we'll set things in motion with connect(), which sets up the QUIC connection, executes the actions list and collects the summary.
let summary = sync_client::connect(config, &actions).expect("connection failed");
println!( "=== received connection summary! ===\n\n{}", serde_json::to_string_pretty(&summary).unwrap_or_else(|e| e.to_string()));</code></pre>
ConnectionSummary provides data about the connection, including the frames h3i received, details about why the connection closed, and connection statistics. The example prints the summary out. However, you can programmatically check it. We do this to write our own internal automation tests.
If you're running the example, it should print something like the following:
Let’s walk through the output. Up first is the StreamMap, which is a record of all frames received on each stream. We can see that we received 5 frames on stream 0: 2 UNKNOWNs, one EnrichedHeaders frame, and two DATA frames.
The UNKNOWN frames are extension frames that are unknown to h3i; the server under test is sending what are known as GREASE frames to help exercise the protocol and ensure clients are not erroring when they receive something unexpected per RFC 9114 requirements.
The EnrichedHeaders frame is essentially an HTTP/3 HEADERS frame, but with some small helpers, like one to get the response status code. The server under test sent a 400 as expected.
The DATA frames carry response body bytes. In this case, the body is the HTML required to render the Cloudflare Bad Request page (you can peek at the HTML yourself in Wireshark). We chose to omit the raw bytes from the ConnectionSummary since they may not be representable safely as text. A future improvement could be to encode the bytes in base64 or hex, in order to support tests that need to check response content.
h3i for test automation
We believe h3i is a great library for building automated tests on. You can take the above example and modify it to fit within various types of (continuous) integration tests.
We outlined earlier how the Protocols team HTTP/3 testing has organically grown to use three different frameworks. Even within those, we still didn't have much flexibility and ease of use. Over the last year we've been building h3i itself and reimplementing our suite of ingress proxy test cases using the Rust library. This has helped us improve test coverage with a range of new tests not previously possible. It also surprisingly identified some problems with the old tests, particularly for some edge cases where it wasn't clear how the old test code implementation was running under the hood.
Bake offs, interop, and wider testing of HTTP
RFC 1025 was published in 1987. Authored by Jon Postel, it discusses bake offs:
In the early days of the development of TCP and IP, when there were very few implementations and the specifications were still evolving, the only way to determine if an implementation was "correct" was to test it against other implementations and argue that the results showed your own implementation to have done the right thing. These tests and discussions could, in those early days, as likely change the specification as change the implementation.
There were a few times when this testing was focused, bringing together all known implementations and running through a set of tests in hopes of demonstrating the N squared connectivity and correct implementation of the various tricky cases. These events were called "Bake Offs".
While nearly 4 decades old, the concept of exercising Internet protocol implementations and seeing how they compare to the specification still holds true. The QUIC WG made heavy use of interoperability testing through its standardization process. We started off sitting in a room and running tests manually by hand (or with some help from scripts). Then Marten Seemann developed the QUIC Interop Runner, which runs regular automated testing and collects and renders all the results. This has proven to be incredibly useful.
The state of HTTP/3 interoperability testing is not quite as mature. Although there are tools such as Kazu Yamamoto's excellent h3spec (in Haskell) for testing conformance, there isn't a similar continuous integration process of collection and rendering of results. While h3i shares similarities with h3spec, we felt it important to focus on the framework capabilities rather than creating a corpus of tests and assertions. Cloudflare is a big fan of Rust and as several teams move to Rust-based proxies, having a consistent ecosystem provides advantages (such as developer velocity).
We certainly feel there is a great opportunity for continued collaboration and cross-pollination between projects in the QUIC and HTTP space. For example, h3i might provide a suitable basis to build another tool (or set of scripts) to run bake offs or interop tests. Perhaps it even makes sense to have a common collection of test cases owned by the community, that can be specialized to the most appropriate or preferred tooling. This topic was recently presented at the HTTP Workshop 2024 by Mohammed Al-Sahaf, and it excites us to see new potential directions of testing improvements.
When using any tools or methods for protocol testing, we encourage responsible handling of security-related matters. If you believe you may have identified a vulnerability in an IETF Internet protocol itself, please follow the IETF's reporting guidance. If you believe you may have discovered an implementation vulnerability in a product, open source project, or service using QUIC or HTTP, then you should report these directly to the responsible party. Implementers or operators often provide their own publicly-available guidance and contact details to send reports. For example, the Cloudflare quiche security policy is available in the Security tab of the GitHub repository.
Summary and outlook
Cloudflare takes testing very seriously. While h3i has a limited feature set as a test HTTP/3 client, we believe it provides a strong framework that can be extended to a wider range of different cases and different protocols. For example, we'd like to add support for low-level HTTP/2.
We've designed h3i to integrate into a wide range of testing methodologies, from manual ad-hoc testing, to native Rust tests, to conformance testbenches built with scripting languages. We've had great success migrating our existing zoo of test tools to a single one that is more accessible and easier to maintain.
Now that you've read about h3i's capabilities, it's left as an exercise to the reader to go back to the example of HTTP/3 control streams and consider how you could write tests to exercise a server.
We encourage the community to experiment with h3i and provide feedback, and propose ideas or contributions to the GitHub repository as issues or Pull Requests.
GitHub’s Top Blogs of 2024https://blog.imc.re/RSSBOX/rss/e38966e3.html2024-12-30T11:47:10.000Z2024-12-30T11:47:10.000Z
As 2024 wraps up, we’re revisiting the highlights of a year packed with innovation, learning, and community. From unlocking the power of AI to mastering essential developer tools, these blogs reflect what makes GitHub the home for all developers. Ready? Let’s jump in.
Choose your own adventure with AI models
GitHub Copilot has long leveraged different large language models (LLMs) for different use cases. Now, you can access these models to meet your own needs.
2024 was all about empowering you to ship better code faster and smarter. Here are the launches that had the community buzzing:
Try GitHub Models. We believe every developer (that’s you!) can be an AI engineer with the right tools and training. Access a number of AI models via a built-in playground that lets you test different prompts and model parameters, for free, right in GitHub. If you like what you’re seeing on the playground, you can easily bring the models to your developer environment in GitHub Codespaces and VS Code and start building.
Unlock more possibilities with GitHub Copilot Extensions. Through a growing partner ecosystem, Copilot Extensions helps you use GitHub Copilot across third-party tools, services, and more (plus you can build your own!), all without leaving the IDE or GitHub.com.
Work smarter with GitHub Copilot Enterprise. Our most advanced AI offering to date is customized to your organization’s knowledge and codebase, infusing GitHub Copilot throughout the software development lifecycle.
Meet GitHub Copilot Workspace, the Copilot-native developer environment where any developer can go from idea to code to software, all in natural language. Sign up for the waitlist today, and we’ll email you when it’s your turn to try Copilot Workspace.
Secure code more than three times faster with Copilot Autofix. Our vision for application security is a world where a vulnerability found means a vulnerability fixed. Copilot Autofix now includes security campaigns to help you remediate vulnerabilities at scale and pay down years of security debt with just a few clicks.
Power faster, more efficient build systems thanks to Arm64 on GitHub Actions. This new addition to our suite of hosted runners provides power, performance, and sustainability improvements for all your GitHub Actions jobs. You can take advantage of Arm-based hardware hosted by GitHub to build and deploy your release assets anywhere Arm architecture is used.
Level up your skills
Keeping up with the always-changing technology landscape requires you to be a lifelong learner. Whether you’re looking to go deep in your current skillset or expand your knowledge to prepare for that next career move, we have resources to help you get there.
The global developer community surged in size in 2024. (Did you know there are now 150M developers on GitHub?!) The Octoverse report highlighted the boom in data science and machine learning: Python overtook JavaScript as the most popular language on GitHub for the first time, and the use of Jupyter Notebooks skyrocketed.
Find out how your favorite tools fared and explore the hottest projects welcoming new contributors. Read the report.
Stay connected
Like what you see? Don’t miss a beat in 2025. Subscribe to our newsletter, GitHub Insider, to keep up with the latest in software development.
We do newsletters, too
Discover tips, technical guides, and best practices in our biweekly newsletter just for developers.
]]>What’s New in Cloudflare: MASQUE Now Powers 1.1.1.1 & WARP Apps, DEX Now Generally Available With Remote Captureshttps://blog.imc.re/RSSBOX/rss/5d0e2abd.html2024-12-27T14:00:00.000Z2024-12-27T14:00:00.000Z
At Cloudflare, we are constantly innovating and launching new features and capabilities across our product portfolio. Today’s roundup blog post shares two exciting updates across our platform: our cross-platform 1.1.1.1 & WARP applications (consumer) and device agents (Zero Trust) now use MASQUE, a cutting-edge HTTP/3-based protocol, to secure your Internet connection. Additionally, DEX is now available for general availability.
Faster and more stable: our 1.1.1.1 & WARP apps now use MASQUE by default
We’re excited to announce that as of today, our cross-platform 1.1.1.1 & WARP apps now use MASQUE, a cutting-edge HTTP/3-based protocol, to secure your Internet connection.
As a reminder, our 1.1.1.1 & WARP apps have two main functions: send all DNS queries through 1.1.1.1, our privacy-preserving DNS resolver, and protect your device’s network traffic via WARP by creating a private and encrypted tunnel to the resources you’re accessing, preventing unwanted third parties or public Wi-Fi networks from snooping on your traffic.
There are many ways to encrypt and proxy Internet traffic — you may have heard of a few, such as IPSec, WireGuard, or OpenVPN. There are many tradeoffs we considered when choosing a protocol, but we believe MASQUE is the future of fast, secure, and stable Internet proxying, it aligns with our belief in building on top of open Internet standards, and we’ve deployed it successfully at scale for customers like iCloud Private Relay and Microsoft Edge Secure Network.
Why MASQUE?
MASQUE is a modern framework for proxying traffic that allows a variety of application protocols, including HTTP/3, to utilize QUIC as their transport mechanism. That’s a lot of acronyms, so let's make sure those are clear.
QUIC is a general-purpose transport protocol and Internet standard that operates on top of UDP (instead of TCP), is encrypted by default, and solves several performance issues that plagued its predecessors. HTTP/3is the latest version of the HTTP protocol, defining the application-layer protocol that runs on top of QUIC as its transport mechanism. MASQUE is a set of mechanisms for tunneling traffic over HTTP. It extends the existing HTTP CONNECT model, to allow tunneling UDP and IP traffic. This is especially efficient when combined with the QUIC’s unreliable datagram extension.
For example, we can use MASQUE’s CONNECT-IP method to establish a tunnel that can send multiple concurrent requests over a single QUIC connection:
The benefit these protocols have for the quality and security of everyone’s Internet browsing experience is real. Earlier transport protocols were built before the advent of smartphones and mobile networks, so QUIC was designed to support a mobile world, maintaining connections even in poorly connected networks, and minimizing disruptions as people switch rapidly between networks as they move through their day. Leveraging HTTP/3 as the application layer means that MASQUE is more like “normal” HTTP traffic on the Internet, meaning that it is easier to support, is compatible with existing firewall and security rules, and that it supports cryptographic agility (i.e. support for post-quantum crypto), making this traffic more secure and resilient in the long term.
Get started now
All new installations of our 1.1.1.1 & WARP apps support MASQUE, including iOS, Android, macOS, Windows, and Linux, and we’ve started to roll it out as the preferred protocol over WireGuard. On mobile, to check if your connection is already secured over MASQUE, or change your device’s default option, you can toggle this setting via Advanced > Connection options > Tunnel protocol:
Protocol connection options shown here on the iOS app
We offer the following options:
Auto: this allows the app to choose the protocol.
MASQUE: always use MASQUE to secure your connection.
WireGuard: always use WireGuard to secure your connection.
On desktop versions, you can switch the protocol by using the WARP command-line interface. For example:
warp-cli tunnel protocol set WireGuardwarp-cli tunnel protocol set MASQUE
With this rollout, we're excited to see MASQUE deliver increased performance and stability to millions of users. Download one of the WARP apps today!
DEX now Generally Available: Announcing detailed device visibility with DEX Remote Captures
Following the successful beta launch of Digital Experience Monitoring (DEX), we are thrilled to announce the general availability of DEX, along with new Remote Captures functionality.
Imagine this: an end user is frustrated with a slow application, and your IT team is struggling to pinpoint the root cause. Traditionally, troubleshooting such issues involved contacting the end user and asking them to manually collect and share network traffic data. This process is time-consuming, prone to errors, and often disruptive to the end user's workflow.
Building upon the capabilities of DEX, we are excited to introduce Remote Captures, a powerful new feature that empowers IT admins to gain unprecedented visibility into end-user devices and network performance. DEX now introduces Remote Captures, a powerful new feature that empowers IT admins to remotely initiate network packet captures (PCAP) and WARP Diag logs directly from your end users’ devices and capture diagnostic information automatically from our device client. This streamlined approach accelerates troubleshooting, reduces the burden on end users, and provides valuable insights into network performance and security.
Why Remote Captures?
Remote Captures offer several key advantages. By analyzing detailed network traffic, IT teams can quickly pinpoint the root cause of network issues. Furthermore, granular network data empowers security teams to proactively detect and investigate potential threats. Finally, by identifying bottlenecks and latency issues, Remote Captures enable organizations to optimize network performance for a smoother user experience.
How Remote Captures work
Initiating a Remote Capture is straightforward. First, select the specific device you wish to troubleshoot. Then, with a few simple clicks, start capturing network traffic and/or WARP Diag data. Once the capture is complete, download the captured data and utilize your preferred tools for in-depth analysis.
Get started today
DEX Remote Captures are now available for Cloudflare One customers. They can be configured by going to Cloudflare Dashboard > Zero Trust > DEX > Remote Captures, and then selecting the device you wish to collect from. For more information, refer to Remote captures. This new capability highlights just one of the many ways our unified SASE platform helps organizations find and fix security issues across SaaS applications. Try it out now using our free tier to get started.
Never miss an update
We hope you enjoy reading our roundup blog posts as we continue to build and innovate. Stay tuned to the Cloudflare Blog for the latest news and updates.
]]>Sometimes I Cache: Implementing Lock-Free Probabilistic Cachinghttps://blog.imc.re/RSSBOX/rss/8a655bdc.html2024-12-26T14:00:00.000Z2024-12-26T14:00:00.000Z
HTTP caching is conceptually simple: if the response to a request is in the cache, serve it, and if not, pull it from your origin, put it in the cache, and return it. When the response is old, you repeat the process. If you are worried about too many requests going to your origin at once, you protect it with a cache lock: a small program, possibly distinct from your cache, that indicates if a request is already going to your origin. This is called cache revalidation.
In this blog post, we dive into how cache revalidation works, and present a new approach based on probability. For every request going to the origin, we simulate a die roll. If it’s 6, the request can go to the origin. Otherwise, it stays stale to protect our origin from being overloaded. To see how this is built and optimised, read on.
Background
Let's take the example of an online image library. When a client requests an image, the service first checks its cache to see if the resource is present. If it is, it returns it. If it is not, the image server processes the request, places the response into the cache for a day, and returns it. When the cache expires, the process is repeated.
Figure 1: Uncached request goes to the origin
Figure 2: Cached request stops at the cache
And this is where things get complex. The image of a cat might be quite popular. Let's say it's requested 10 times per second. Let’s also assume the image server cannot handle more than 1 request per second. After a day, the cache expires. 10 requests hit the service. Given there are no up-to-date items in cache, these 10 requests are going to go directly to the image server. This problem is known as cache stampede. When the image server sees these 10 requests all happening at the same time, it gets overloaded.
Figure 3: Image server overloaded upon cache expiration. This can happen to one or multiple users, across locations.
This all stops if the cache gets populated, as it can handle a lot more requests than the origin.
Figure 4: Cache is populated and can handle the load. The image server is healthy again.
In the following sections, we build this image service, see how it can prevent cache stampede with a cache lock, then dive into probabilistic cache revalidation, and its optimisation.
Setup
Let's write this image service. We need an image, a server, and a cache. For the image we're going to use a picture of my cat, Cloudflare Workers for the server, and the Cloudflare Cache API for caching.
Note to the reader: On purpose, we aren’t using Cloudflare KV or Cloudflare CDN Cache, because they already solve our cache validation problem by using a cache lock.
let cache = caches.defaultconst CACHE_KEY = new Request('https://cache.local/')const CACHE_AGE_IN_S = 86_400 // 1 day
function cacheExpirationDate() { return new Date(Date.now() + 1000*CACHE_AGE_IN_S) }
function fetchAndCache(ctx) { let response = await fetch(‘https://files.research.cloudflare.com/images/cat.jpg') response = new Response( await response.arrayBuffer(), { headers: { ‘Content-Type’: response.headers.get(‘Content-Type’), ‘Expires’: cacheExpirationDate().toUTCString(), }, }, ) ctx.waitUntil(cache.put(CACHE_KEY, response.clone())) return response }
Codeblock 1: Image server with a non-collapsing cache
Expectation about cache revalidation
The image service is receiving 10 requests per second, and it caches images for a day. It's reasonable to assume we would like to start revalidating the cache 5 minutes before it expires. The code evolves as follows:
let cache = caches.defaultconst CACHE_KEY = new Request('https://cache.local/')const CACHE_AGE_IN_S = 86_400 // 1 dayconst CACHE_REVALIDATION_INTERVAL_IN_S = 300
function cacheExpirationDate() { // Date constructor in workers takes Unix time in milliseconds // Date.now() returns time in milliseconds as well return new Date(Date.now() + 1000*CACHE_AGE_IN_S) }
async function fetchAndCache(ctx) { let response = await fetch(‘https://files.research.cloudflare.com/images/cat.jpg') response = new Response( await response.arrayBuffer(), { headers: { ‘Content-Type’: response.headers.get(‘Content-Type’), ‘Expires’: cacheExpirationDate().toUTCString(), }, }, ) ctx.waitUntil(cache.put(CACHE_KEY, response.clone())) return response }
// Revalidation function added here // This is were we are going to focus our effort: should the request be revalidated ? function shouldRevalidate(expirationDate) { let remainingCacheTimeInS = (expirationDate.getTime() - Date.now()) / 1000
export default { async fetch(request, env, ctx) { let cachedResponse = await cache.match(CACHE_KEY) if (cachedResponse) { // revalidation happens only if the request was cached. Otherwise, the resource is fetched anyway if (shouldRevalidate()) { ctx.waitUntil(fetchAndCache(ctx)) } return cachedResponse } return fetchAndCache(ctx) } }
Codeblock 2: Image server with early-revalidation and a non-collapsing cache
That code works, and we can now revalidate 5 minutes in advance of cache expiration. However, instead of fetching the image from the origin server at expiration time, all requests are going to be made 5 minutes in advance, and that does not solve our cache stampede problem. This happens no matter if requests are coming to a single location or not, given the code above does not collapse requests.
To solve our cache stampede problem, we need the revalidation process to not send too many requests at the same time. Ideally, we would like only one request to be sent between expiration - 5min and expiration.
The usual solution: a cache lock
To make sure there is only one request at a time going to the origin server, the solution that's usually deployed is a cache lock. The idea is that for a specific item, a cat picture in our case, requests to the origin try to obtain a lock. The request obtaining the lock can go to the origin, the others will serve stale content.
The lock has two methods: try_lock() and unlock.* try_lock if the lock is free, take it and return true. If not, return false.* unlock releases the lock.
Codeblock 3: Lock service implemented with a Durable Object
That service can then be used as a cache lock.
// CACHE_LOCK is an instantiation of the above binding// Assuming the above is deployed as a worker with name `lock`// It can be bound in wrangler.toml as follows// services = [ { binding = "CACHE_LOCK", service = "lock" } ]
const LOCK_KEY = “cat_image_service”
async function fetchAndCache(env, ctx) { let response = await fetch(‘…’) ctx.waitUntil(env.CACHE_LOCK.unlock(LOCK_KEY)) … }
function shouldRevalidate(env, expirationDate) { let remainingCacheTimeInS = (expirationDate.getTime() - Date.now()) / 1000
// check if the expiry window is now, and then if the revalidation lock is available. if it is, take it return remainingCacheTimeInS <= CACHE_REVALIDATION_INTERVAL_IN_S && env.CACHE_LOCK.try_lock(LOCK_KEY) }
Codeblock 4: Image server with early-revalidation and a cache using a cache-lock
Now you might say "Et voilà. No need for probabilities and mathematics. Peak engineering has triumphed." And you might be right, in most cases. That's why cache locks are so predominant: they are conceptually simple, deterministic for the same key, and scale well with predictable resource usage.
On the other hand, cache locks add latency and fallibility. To take ownership of a lock, cache revalidation has to contact the lock service. This service is shared across different processes, possibly different machines in different locations. Requests therefore take time. In addition, this service might be unavailable. Probabilistic cache revalidation does not suffer from these, given it does not reach out to an external service but rolls a die with the local randomness generator. It does so at the cost of not guaranteeing the number of requests going to the origin server: maybe zero for an extended period, maybe more than one. On average, this is going to be fine. But there can be border cases, similar to how one can roll a die 10 times and get 10 sixes. It’s unlikely, but not unrealistic, and certain services need that certainty. In the following sections, we dissect this approach.
First dive into probabilities given a stable request rate
A first approach is to reduce the number of requests going to the origin server. Instead of always sending a request to revalidate, we are going to send 1 out of 10. This means that instead of sending 10 requests per second when the cache is invalidated, we send 1 per second.
Because we don't have a lock, we do that with probabilities. We set the probability of sending a request to the origin to be $p=\frac{1}{10}$. With a rate of 10 requests per second, after 1 second, the expectancy of a request being sent to the origin is $1-(1-p)^10=65\%$. We draw the evolution of the function $E(r, t)=1-(1-p)^{r \times t}$ representing the expectancy of a request being sent to the server over time. $r = 10$ and is the request rate.
Figure 5: Revalidation time $E(t)$ with $r=10$ and $p=\frac{1}{10}$. At time $t$, $E(t)$ is the probability that an early revalidation occurred.
The graph moves very quickly towards $1$. This means we might still have space to reduce the number of requests going to our origin server. We can set a lower probability, such as $p_2=\frac{1}{500}$ (1 request every 5 seconds on average). The graph looks as follows:
Figure 6: Revalidation time $E(t)$ with $r=10$ and $p=\frac{1}{500}$.
Codeblock 5: Image server with early-revalidation and a probabilistic cache using uniform distribution
That's it. If the cache is not close to expiration, we don't revalidate. If the cache is expired, we revalidate. Otherwise, we revalidate based on a probability.
Adaptive cache revalidation
Until now, we assumed the picture of the cat received a stable request rate. However, for a real service, this does not necessarily hold. For instance, if instead of 10 requests per second, imagine the service receives only 1. The expectancy function does not look as good. After 5 minutes (300s), $E(r=1, t=300)=45\%$. On the other hand, if the image service is receiving 10,000 requests per second, $E(r=10000, t = 300) \approx 100\%$, but our server receives on average $10000 \times \frac{1}{500} = 20$ requests per second. It would be ideal to design a probability function that would adapt to the request rate.
That function would return a low probability when expiration time is far in the future, and increase over time such that the cache is revalidated before it expires. It would cap the request rate going to the origin server.
Let’s design the variation of probability $p$ over 5 minutes. When far from the expiration, the probability to revalidate should be low. This should help match the high request rate. For example, with a request rate of 10k requests per second, we would like the revalidation probability $p$ to be $\frac{1}{100000}$. This ensures the request rates seen by our server are going to be low on average, at about 1 request every 10 seconds. As time passes, we increase this probability to allow for revalidation even at a lower request rate.
Time to expiration $t$ (in s)
Revalidation probability $p$
Target request rate $r$ (in rps)
300
1/100000
10000
240
1/10000
1000
180
1/1000
100
120
1/100
10
60
1/10
1
0
1
-
Table 1: Variation of revalidation probability over time
For each of these intervals, there is a high likelihood that a request rate $r$ will trigger a cache revalidation, and low likelihood that a lower request rate will trigger it. If it does, it's ok.
We can update our revalidation function as follows:
function shouldRevalidate(expirationDate) { let remainingCacheTimeInS = (expirationDate.getTime() - Date.now()) / 1000
if (remainingCacheTimeInS > CACHE_REVALIDATION_INTERVAL_IN_S) { return false } if (remainingCacheTimeInS <= 0) { return true } let currentMinute = Math.floor(remainingCacheTimeInS/60) return Math.random() < CACHE_REVALIDATION_PROBABILITY_PER_MIN[currentMinute] }
Codeblock 6: Image server with early-revalidation and a probabilistic cache using piecewise uniform distribution
Optimal cache stampede solution
There seems to be a lot of decisions going on here. To solve this, we can reference an academic paper written by A Vattani, T Chierichetti, and K Lowenstein in 2015 called Optimal Probabilistic Cache Stampede Prevention. If you read it, you'll recognise that what we have been discussing until now is close to what the paper presents. For instance, both the cache revalidation algorithm structure and the early revalidation function look similar.
Figure 7: Probabilistic early expiration of a cache item as defined by Figure 2 of Optimal Probabilistic Cache Stampede Prevention paper. In our case, $\mathcal{D}=300$
One takeaway from the paper is that instead of discretization, with a probability from 0 to 60s, then from 60s to 120s, …, the probability function can be continuous. Instead of a fixed $p$, there is a function $p(t)$ of time $t$.
$p(t)=e^{-\lambda (expiry-t)}, \text{ with } expiry=300, \text{ and } t \in [0, 300]$
We call $\lambda$ the steepness parameter, and set it to $\frac{1}{300}$, $300$ being our early expiration gap.
The expectancy over time is $E(r, t)=1-e^{-rλt}$. This leads to the expectancy below for various request rates. You can note that when $r=1$, there is not a $100%$ chance that the request will be revalidated before expiry.
Figure 8: Revalidation time $E(t)$ for multiple $r$ with an exponential distribution.
function shouldRevalidate(expirationDate) { let remainingCacheTimeInS = (expirationDate.getTime() - Date.now()) / 1000
if (remainingCacheTimeInS > CACHE_REVALIDATION_INTERVAL_IN_S) { return false } if (remainingCacheTimeInS <= 0) { return true } // p(t) is evaluated here return Math.random() < Math.exp(-REVALIDATION_STEEPNESS*remainingCacheTimeInS) }
Codeblock 7: Image server with early-revalidation and a probabilistic cache using exponential distribution
And that's it. Given Date.now() has a granularity, and is not continuous, it would also be possible to discretise these functions, even though the gains are minimal. This is what we have done in a production worker implementation, where the number of requests is important. It is a service that benefits from caching for performance consideration, and that cannot use built-in stale-while-revalidate from within Cloudflare workers. Probabilistic cache stampede prevention is well-suited here, as no new component has to be built, and it performs well at different request rates.
Conclusion
We have seen how to solve cache stampede without a lock, its implementation, and why it is optimal. In the real world, you likely will not encounter this issue: either because it’s good enough to optimize your origin service to serve more requests, or because you can leverage a CDN cache. In fact, most HTTP caches provide an API that follows Cache Control, and likely have all the tools you need. This primitive is also built into certain products, such as Cloudflare KV.
If you have not done so, you can go and experiment with all the code snippets presented in this blog on the Cloudflare Workers Playground at cloudflareworkers.com.
]]>Announcing CodeQL Community Packshttps://blog.imc.re/RSSBOX/rss/d4d7562.html2024-12-26T06:17:51.000Z2024-12-26T06:17:51.000Z
We are excited to introduce the new CodeQL Community Packs, a comprehensive set of queries and models designed to enhance your code analysis capabilities. These packs are tailored to augment the standard set of CodeQL queries, providing additional resources for security researchers and developers alike.
Why?
CodeQL is a semantic code analysis tool that allows developers to query their codebases as databases, enabling the identification of vulnerabilities, bugs, and patterns efficiently.
The standard set of CodeQL queries is focused on accuracy and low false positive rates, which is ideal for integration into CI/CD pipelines where alerts are primarily handled by developers. However, when alerts are operated by security engineers or researchers, the balance between false positives and false negatives can be adjusted to prioritize low false negatives, ensuring no bugs are left behind—albeit at the cost of more triaging.
What?
The CodeQL Community Packs is a set of CodeQL packs to augment the standard set queries. They include three main types of packs:
Model packs: these packs contain additional models of Taint Tracking sources, sinks, and summaries for libraries and frameworks that are not supported by the default suites.
Query packs: these packs contain additional security and audit queries to help identify potential vulnerabilities and improve code quality.
Library packs: designed to be used by query packs, these packs do not contain queries themselves but provide essential libraries for more comprehensive analysis.
How?
The GitHub Security Lab has been extensively using these packs for the last few years and as our records show, they turned out to be very fruitful.
In addition to the additional queries and models provided by the community packs, we have also been using the audit queries, which proved invaluable when running deep-dive manual code reviews, such as the ones we did for Datahub and Home Assistant. Being able to list all the files which introduced untrusted data into the application or that perform security-relevant operations was really helpful when exploring unfamiliar huge codebases, such as the Home Assistant one.
What’s in the community packs?
The CodeQL Community Packs offer a variety of additional queries and models for languages, such as Java, C#, and Python. These packs are designed to move the Signal to Noise (SNR) ratio closer to the low false negatives end of the spectrum, making them particularly useful for security researchers.
Security: dozens of new security queries contributed by CodeQL engineers and security researchers from the GitHub Security Lab, but also by the broader community of security researchers.
Audit Exploration: queries to list all files, dependencies, untrusted data entry points, and hazardous sinks.
Audit Templates: templates to build your own taint tracking queries, explore data paths, or “hoist” sinks to public method parameters.
Library sources: special queries designed to find third-party APIs called with untrusted data.
Additional threat model pack which defines library API method parameters as a source of untrusted data.
Library extension models
Remember Log4Shell? It was relatively easy for a SAST tool to detect, as the JNDI injection sink was well-known and covered by existing CodeQL models at that time. However, CodeQL’s default threat model, like most SAST tools, is based on modeling untrusted data as data that comes from the network. Therefore, CodeQL could have reported Log4Shell if we had analyzed an application that took untrusted data from the network (for example, a web application) and passed this untrusted data to Log4J logger methods.
To enable CodeQL to report such a data flow path, we would have needed to provide CodeQL with the source code of both the web application and Log4J. Could we have reported Log4Shell by analyzing only the Log4J source code? Certainly! But we would have needed a different threat model, one in which the arguments of logger methods such as info or error were considered sources of untrusted data. But how could CodeQL know that these methods could introduce untrusted data in the first place?
To support such a threat model, we developed the library source packs. We analyzed thousands of applications that took untrusted data and passed it to third-party APIs (such as Log4J’s error method). This analysis resulted in a list of third-party library methods used in real applications that are passed untrusted data.
Once we collected this list, which contained API methods such as Log4J’s AbstractLogger.error, we used it to define new sources of untrusted data to be used when scanning library code, such as Log4J code. By doing this with Log4J code, we were able to first identify that logger methods can be called with untrusted data from network requests and second, report a JNDI injection in Log4J code when using the new library source QL packs!
Exploration queries
Reviewing a new, unfamiliar codebase is a difficult and lengthy process. Reducing the review surface to the most significant and relevant files is crucial to making this process as efficient as possible.
When faced with similar reviews, the GitHub Security Lab likes to first map out the new codebase. We do this by listing all the entry points where potentially untrusted data enters the application and identifying operations that can be hazardous, such as file reads/writes, deserialization operations, or network requests.
To achieve this, we use the RemoteFlowSources.ql query, which provides a list of all places identified by CodeQL where untrusted data enters the application. We also use the HotSpots query, which returns a list of all hazardous sinks in the application, regardless of evidence of untrusted data flowing into them.
In addition to providing a good initial heat map of the codebase, this approach helps us better understand how well CodeQL covers the used libraries and whether additional modeling is needed.
How to use them?
The community packs are regular CodeQL packs and can be used both as part of GitHub’s code scanning workflows and with the CodeQL CLI.
To use the CodeQL community packs in code scanning, specify a with: packs: entry in the uses: github/codeql-action/init@v3 section of your CodeQL code scanning workflow. See the examples below.
Adding the community packs library extension models to a scan:
The most important aspect of the community packs is the community involvement! Sharing your models and queries with the community is the best way to help secure the open source software we all depend on. Contributions can range from simple Model As Data (MaD) lines to existing extension files or even the creation of new queries that model new vulnerability classes. Every contribution is welcome!
]]>Un Experimento Rápido: Translating Cloudflare Stream Captions With Workers AIhttps://blog.imc.re/RSSBOX/rss/302a2768.html2024-12-24T14:00:00.000Z2024-12-24T14:00:00.000Z
Cloudflare Stream launched AI-powered automated captions to transcribe English in on-demand videos in March 2024. Customers' immediate next questions were about other languages — both transcribing audio from other languages, and translating captions to make subtitles for other languages. As the Stream Product Manager, I've thought a lot about how we might tackle these, but I wondered…
What if I just translated a generated VTT (caption file)? Can we do that? I hoped to use Workers AI to conduct a quick experiment to learn more about the problem space, challenges we may find, and what platform capabilities we can leverage.
There is a sample translator demo in Workers documentation that uses the “m2m100-1.2b” Many-to-Many multilingual translation model to translate short input strings. I decided to start there and try using it to translate some of the English captions in my Stream library into Spanish.
Selecting test content
I started with my short demo video announcing the transcription feature. I wanted a Worker that could read the VTT captions file from Stream, isolate the text content, and run it through the model as-is.
The first step was parsing the input. A VTT file is a text file that contains a sequence of numbered “cues,” each with a number, a start and end time, and text content.
1 00:00:00.000 –> 00:00:02.580 Good morning, I’m Taylor Smith,
2 00:00:02.580 –> 00:00:03.520 the Product Manager for Cloudflare
3 00:00:03.520 –> 00:00:04.460 Stream. This is a quick
4 00:00:04.460 –> 00:00:06.040 demo of our AI-powered automatic
5 00:00:06.040 –> 00:00:07.580 subtitles feature. These subtitles
6 00:00:07.580 –> 00:00:09.420 were generated with Cloudflare WorkersAI
7 00:00:09.420 –> 00:00:10.860 and the Whisper Model,
8 00:00:10.860 –> 00:00:12.020 not handwritten, and it took
9 00:00:12.020 –> 00:00:13.940 just a few seconds.
Parsing the input
I started with a simple Worker that would fetch the VTT from Stream directly, run it through a function I wrote to deconstruct the cues, and return the timestamps and original text in an easier to review format.
// Step Two: Parse the VTT file and get the textconst captions = vttToCues(input);// Done: Return what we have.return new Response(captions.map(c => (`#${c.number}: ${c.start} --> ${c.end}: ${c.content.toString()}`)).join('\n'));
}, };
That returned this text:
#1: 0 --> 2.58: Good morning, I'm Taylor Smith,#2: 2.58 --> 3.52: the Product Manager for Cloudflare#3: 3.52 --> 4.46: Stream. This is a quick#4: 4.46 --> 6.04: demo of our AI-powered automatic#5: 6.04 --> 7.58: subtitles feature. These subtitles#6: 7.58 --> 9.42: were generated with Cloudflare WorkersAI#7: 9.42 --> 10.86: and the Whisper Model,#8: 10.86 --> 12.02: not handwritten, and it took#9: 12.02 --> 13.94: just a few seconds.
AI-ify
As a proof of concept, I adapted a snippet from the demo into my Worker. In the example, the target language and input text are extracted from the user’s request. In my experiment, I decided to hardcode the languages. Also, I had an array of input objects, one for each cue, not just a string. After interpreting the caption input but before returning a response, I used a map callback to parallelize all the AI.run() calls to translate each cue, so they could execute asynchronously and in-place, then awaited them all to resolve. Ultimately, the AI inference call itself is the simplest part of the script.
Then the script returns the translated output in the format from before.
Of course, this is not a scalable or error-tolerant approach for production use because it doesn’t make affordances for rate limiting, failures, or processing bigger throughput. But for a few minutes of tinkering, it taught me a lot.
#1: 0 --> 2.58: Buen día, soy Taylor Smith.#2: 2.58 --> 3.52: El gerente de producto de Cloudflare#3: 3.52 --> 4.46: Rápido, esto es rápido#4: 4.46 --> 6.04: La demostración de nuestro automático AI-powered#5: 6.04 --> 7.58: Los subtítulos, estos subtítulos#6: 7.58 --> 9.42: Generado con Cloudflare WorkersAI#7: 9.42 --> 10.86: y el modelo de susurro,#8: 10.86 --> 12.02: No se escribió, y se tomó#9: 12.02 --> 13.94: Sólo unos segundos.
A few immediate observations: first, these results came back surprisingly quickly and the Workers AI code worked on the first try! Second, evaluating the quality of translation results is going to depend on having team members with expertise in those languages. Because — third, as a novice Spanish speaker, I can tell this output has some issues.
Cues 1 and 2 are okay, but 3 is not (“Fast, this is fast” from “[Cloudflare] Stream. This is a quick…”). Cues 5 through 9 had several idiomatic and grammatical issues, too. I theorized that this is because Stream splits the English captions into groups of 4 or 5 words to make them easy to read quickly in the overlay. But that also means sentences and grammatical constructs are interrupted. When those fragments go to the translation model, there isn’t enough context.
Consolidating sentences
I speculated that reconstructing sentences would be the most effective way to improve translation quality, so I made that the one problem I attempted to solve within this exploration. I added a rough pre-processor in the Worker that tries to merge caption cues together and then splits them at sentence boundaries instead. In the process, it also adjusts the timing of the resulting cues to cover the same approximate timeframe.
Looking at each cue in order:
// Break this cue up by sentence-ending punctuation.const sentences = thisCue.content.split(/(?<=[.?!]+)/g);
// Cut here? We have one fragment and it has a sentence terminator. const cut = sentences.length === 1 && thisCue.content.match(/[.?!]/);
But if there’s a cue that splits into multiple sentences, cut it up and split the timing. Leave the final fragment to roll into the next cue:
else if (sentences.length > 1) { // Save the last fragment for later const nextContent = sentences.pop();
// Put holdover content and all-but-last fragment into the content newContent += ‘ ‘ + sentences.join(‘ ‘);
result.push({ number: newNumber, start: newStart, end: thisCue.start + (thisLength / 2), // End this cue early content: newContent,});// … then treat the next cue as a holdovercueLength = 1;newContent = nextContent;// Start the next consolidated cue halfway into this cue's original durationnewStart = thisCue.start + (thisLength / 2) + 0.001;// Set the next consolidated cue's number to this cue's numbernewNumber = thisCue.number;
} }
Applying that to the input, it generates sentence-grouped output, visualized here in green:
There are only 3 “new” cues, each starts at the beginning of a sentence. The consolidated cues are longer and might be harder to read when overlaid on a video, but they are complete grammatical units:
#1: 0 --> 3.755: Good morning, I'm Taylor Smith, the Product Manager for Cloudflare Stream.#3: 3.756 --> 6.425: This is a quick demo of our AI-powered automatic subtitles feature.#5: 6.426 --> 12.5: These subtitles were generated with Cloudflare Workers AI and the Whisper Model, not handwritten, and it took just a few seconds.
Translating this “prepared” input the same way as before:
#1: 0 --> 3.755: Buen día, soy Taylor Smith, el gerente de producto de Cloudflare Stream.#3: 3.756 --> 6.425: Esta es una demostración rápida de nuestra función de subtítulos automáticos alimentados por IA.#5: 6.426 --> 12.5: Estos subtítulos fueron generados con Cloudflare WorkersAI y el Modelo Whisper, no escritos a mano, y solo tomó unos segundos.
¡Mucho mejor! [Much better!]
Re-exporting to VTT
To use these translated captions on a video, they need to be formatted back into a VTT with renumbered cues and properly formatted timestamps. Ultimately, the solution should automatically upload them back to Stream, too, but that is an established process, so I set it aside as out of scope. The final VTT result from my Worker is this:
WEBVTT
1 00:00:00.000 –> 00:00:03.754 Buen día, soy Taylor Smith, el gerente de producto de Cloudflare Stream.
2 00:00:03.755 –> 00:00:06.424 Esta es una demostración rápida de nuestra función de subtítulos automáticos alimentados por IA.
3 00:00:06.426 –> 00:00:12.500 Estos subtítulos fueron generados con Cloudflare WorkersAI y el Modelo Whisper, no escritos a mano, y solo tomó unos segundos.
I saved it to a file locally and, using the Cloudflare Dashboard, I added it to the video which you may have noticed embedded at the top of this post! Captions can also be uploaded via the API.
More testing and what I learned
I tested this script on a variety of videos from many sources, including short social media clips, 30-minute video diaries, and even a few clips with some specialized vocabulary. Ultimately, I was surprised at the level of prototype I was able to build on my first afternoon with Workers AI. The translation results were very promising! In the process, I learned a few key things that I will be bringing back to product planning for Stream:
We have the tools. Workers AI has a model called "m2m100-1.2b" from Hugging Face that can do text translations between many languages. We can use it to translate the plain text cues from VTT files — whether we generate them or they are user-supplied. We’ll keep an eye out for new models as they are added, too.
Quality is prone to "copy-of-a-copy" effect. When auto-translating captions that were auto-transcribed, issues that impact the English transcription have a huge downstream impact on the translation. Editing the source transcription improves quality a lot.
Good grammar and punctuation counts. Translations are significantly improved if the source content is grammatically correct and punctuated properly. Punctuation is often missing when captions are auto-generated, but not always — I would like to learn more about how to predict that and if there are ways we can increase punctuation in the output of transcription jobs. My cue consolidator experiment returns giant walls of text if there’s no punctuation on the input.
Translate full sentences when possible. We split our transcriptions into cues of about 5 words for several reasons. However, this produces lower quality output when translated because it breaks grammatical constructs. Translation results are better with full sentences or at least complete fragments. This is doable, but easier said than done, particularly as we look toward support for additional input languages that use punctuation differently.
We will have blind spots when evaluating quality. Everyone on our team was able to adequately evaluate English transcriptions. Sanity-checking the quality of translations will require team members who are familiar with those languages. We state disclaimers about transcription quality and offer tips to improve it, but at least we know what we're looking at. For translations, we may not know how far off we are in many cases. How many readers of this article objected to the first translation sample above?
Clear UI and API design will be important for these related but distinct workflows. There are two different flows being requested by Stream customers: "My audio is in English, please make translated subtitles" alongside "My audio is in another language, please transcribe captions as-is." We will need to carefully consider how we shape user-facing interactions to make it really clear to a user what they are asking us to do.
Workers AI is really easy to use. Sheepishly, I will admit: although I read Stream's code for the transcription feature, this was the first time I've ever used Workers AI on my own, and it was definitely the easiest part of this experiment!
Finally, as a product manager, it is important I remain focused on the outcome. From a certain point of view, this experiment is a bit of an XY Problem. The need is "I have audio in one language and I want subtitles in another." Are there other avenues worth looking into besides "transcribe to captions, then restructure and translate those captions?" Quite possibly. But this experiment with Workers AI helped me identify some potential challenges to plan for and opportunities to get excited about!
]]>When the World Logs Off: Christmas, New Year’s, and the Internet’s Holiday Rhythmhttps://blog.imc.re/RSSBOX/rss/ce9d603a.html2024-12-24T10:00:00.000Z2024-12-24T10:00:00.000Z
As January approaches and the year comes to a close, distinct changes in global Internet usage emerge. Year-end traditions — ranging from Christmas feasts to New Year’s Eve (NYE) countdowns — shape online behavior across continents and cultures. Looking back at Christmas and NYE 2023 offers insights into how these trends may repeat this year, and by January 2025, we’ll be able to directly compare patterns. Examining data from 50 countries and regions reveals how people celebrated in 2023-2024, providing a timely reminder of typical holiday trends.
With Cloudflare’s global reach, we observe planet-wide and local Internet habits during the holiday season. In the past, unintended trends during Christmas and New Year’s Eve have surfaced through our Outage Center, which uses automatic traffic anomaly alerts to detect Internet outages or unusual patterns. In the 2023 overview below, traffic dropped enough on those days to trigger dozens of anomaly alerts (orange and pink bubbles):
While Christmas dominates in many regions, other cultural and religious holidays — like Hanukkah or regional festivities — shape online rhythms in places where Western traditions are less central.
In countries and regions where Christmas is deeply rooted, Internet traffic slows during Christmas Eve dinners, midnight masses, morning gift exchanges, and Christmas Day lunches.
This blog post focuses exclusively on non-bot-related Internet traffic requests, filtering out automated activity to provide a clearer view of genuine human behavior during the holiday season. Before going into location-specific perspectives, here’s a global hourly view of Internet traffic during Christmas and New Year’s Eve 2023 from Cloudflare Radar Data Explorer, highlighting notable drops (in UTC, so it captures impacts across more days due to time zones spanning over 23 hours, from New Zealand to Hawaii in the US):
Global Christmas and New Year’s Eve daily trends
Let’s start with a ranking of countries and regions and their top low-traffic holiday dates, showing each day’s percentage drop. Many locations like the US see clear dips on December 24 and 25 as people celebrate Christmas Eve and Christmas Day offline, and some show smaller declines (compared to Christmas) around December 31 as the New Year approaches. Still, the exact order and magnitude differ, reflecting cultural nuances — some nations experience greater drops on Christmas Eve, others on Christmas Day, and others signal unique patterns tied to New Year’s Eve or January 1 celebrations.
In the next table, locations are listed first (in the left column) by those with the lowest traffic on December 24 (and the highest percentage of traffic drop), followed by December 25, and finally December 31 (in the right column).
Top days with the lowest Internet traffic in December 2023 - January 2024
(with respective percentage drops, if any, from the previous week)
Denmark
#1 December 24 (-35%)
#2 December 25 (-11%)
#3 December 31
South Africa
#1 December 25 (-27%)
#2 December 24 (-15%)
#3 December 31 (-5%)
Norway
#1 December 24 (-32%)
#2 December 25 (-12%)
#3 December 31
United Kingdom
#1 December 25 (-26%)
#2 December 24 (-19%)
#3 December 31
Portugal
#1 December 24 (-32%)
#2 December 25 (-24%)
#3 December 31
Italy
#1 December 25 (-25%)
#2 December 24 (-25%)
#3 December 31
Poland
#1 December 24 (-31%)
#2 December 25 (-21%)
#3 December 31
Australia
#1 December 25 (-25%)
#2 December 24 (-15%)
#3 December 31 (-1%)
Spain
#1 December 24 (-28%)
#2 December 25 (-25%)
#3 December 31
Ireland
#1 December 25 (-24%)
#2 December 24 (-22%)
#3 December 23
Sweden
#1 December 24 (-26%)
#2 December 25 (-6%)
#3 December 31
New Zealand
#1 December 25 (-22%)
#2 December 24 (-8%)
#3 December 31 (-4%)
Chile
#1 December 24 (-23%)
#2 December 25 (-24%)
#3 December 31 (-3%)
Canada
#1 December 25 (-19%)
#2 December 24 (-15%)
#3 December 31
Finland
#1 December 24 (-23%)
#2 December 25 (-16%)
#3 December 31
Nigeria
#1 December 25 (-18%)
#2 December 24 (-19%)
#3 January 1
France
#1 December 24 (-22%)
#2 December 25 (-19%)
#3 December 23
Philippines
#1 December 25 (-16%)
#2 December 24 (-7%)
#3 December 31
Germany
#1 December 24 (-21%)
#2 December 25 (-9%)
#3 December 31
Hong Kong
#1 December 25 (-9%)
#2 December 24 (-6%)
#3 December 23
Mexico
#1 December 24 (-21%)
#2 December 25 (-19%)
#3 December 31
Belgium
#1 December 31 (-1%)
#2 December 24 (-20%)
#3 December 25 (-17%)
Belgium
#1 December 24 (-20%)
#2 December 25 (-17%)
#3 December 31 (-1%)
Indonesia
#1 December 31 (-1%)
#2 December 25 (-7%)
#3 December 24
Romania
#1 December 24 (-20%)
#2 December 25 (-14%)
#3 December 31 (-3%)
Netherlands
#1 December 31 (-10%)
#2 December 24 (-10%)
#3 December 25 (-20%)
United States
#1 December 24 (-16%)
#2 December 25 (-21%)
#3 December 31
Ukraine
#1 December 31 (-10%)
#2 December 24 (-5%)
#3 December 30
Brazil
#1 December 24 (-14%)
#2 December 25 (-26%)
#3 December 31
Thailand
#1 December 31 (-6%)
#2 January 1 (-2%)
#3 December 25 (-2%)
Colombia
#1 December 24 (-14%)
#2 December 25 (-26%)
#3 December 31 (-4%)
The data shows that in many European countries — such as Denmark, Norway, the United Kingdom, Portugal, Italy, Poland, Spain, Ireland, Sweden, Finland, France, Germany, Belgium, the Netherlands, and Romania — Christmas Eve (December24) and Christmas Day (December25) consistently register the biggest drops in Internet traffic. These dips suggest that in much of Europe, Christmas traditions take people firmly offline, whether it’s for family gatherings, festive meals, or religious observances. Outside Europe, similar patterns appear in predominantly Christian-influenced regions, including Australia, New Zealand, Canada, the United States, and several Latin American countries (like Brazil, Chile, and Colombia), confirming that the holiday’s cultural importance is mirrored in their online habits.
In contrast, locations less influenced by Western Christmas traditions, such as those in Asia, show subtler or different patterns. For example, Hong Kong and the Philippines do show declines in traffic, reflecting a hybrid of local and global influences, while places like Thailand and Indonesia present smaller dips on Christmas compared to other days or emphasize different holidays altogether. These variations highlight that while Christmas exerts a strong pull offline in many parts of the world, its impact on Internet usage is shaped by local cultural contexts.
As an example, here’s the US perspective from Cloudflare Radar Data Explorer, where the drop in traffic during Christmas and New Year 2023 is evident:
Where Christmas isn’t central
Not every country’s December revolves around Christmas. In Israel, for example, Hanukkah’s timing changes year to year, influencing when people log off. In 2023, Hanukkah started on December 7, leading to an 8% traffic drop that day and 7% on the following days through December 10. Interestingly, in some years like 2024, Hanukkah begins closer to December 25, potentially overlapping with Western Christmas.
Countries where Christmas didn’t have a clear impact
Turkey
#1 December 31 (-18%)
#2 December 29
#3 December 30
Israel
#1 December 29
#2 January 5
#3 December 30
Japan
#1 December 31 (-8%)
#2 January 1
#3 December 30 — December 24 with -3%
Vietnam
#1 January 1 (-7%)
#2 December 31 (-3%)
#3 January 2
Russia
#1 December31 (-23%)
#2 January 1 (-15%)
#3 December 30
Singapore
#1 December 16
#2 December 17
#3 December 18
India
#1 December 17
#2 December 16
#3 December 24
Bangladesh
#1 December 15
#2 December 16
#3 December 18
Saudi Arabia
#1 January 5
#2 January 6
#3 January 8
China
#1 December 19
#2 December 15
#3 December 18
Now, let’s focus on a more granular perspective of these trends, showing the impact of Christmas dinners and lunches, and also New Year’s Eve drops in traffic.
Note: Unless otherwise noted, all times used in this blog post are local ones; in countries with several timezones, we’re using the timezone where more people live (for the US, Eastern time is used).
A more granular perspective of Christmas: offline feasts and morning quiet
Europe
In Europe, Christmas traditions dominate, leading to the most significant Internet traffic drops. Christmas Eve dinner is a near-universal offline moment, with countries like Spain (-70% at 21:45), Portugal (-70% at 20:30), and Denmark (-68% at 19:45) experiencing the steepest declines. On Christmas Day, mornings are quieter as people relax or attend religious services, while festive lunches drive further drops, with traffic down 43% at 13:45 in Portugal and 44% at 07:15 in France.
By Boxing Day (December 26), digital activity rebounds as people return online for sales or socializing. For instance, the UK shows a 16% increase at 13:00, while Canada records a 19% rise at 08:15. In Australia, traffic climbs by 20% at 09:30, illustrating regional differences in how the day is celebrated.
Americas
In the Americas, holiday patterns reflect a mix of cultural traditions. In the United States, Christmas Eve traffic drops by 29% at 20:15, aligning with evening family gatherings, and Christmas Day sees a 32% decline at 09:15, reflecting quieter mornings.
In Latin America, Christmas Eve (Nochebuena) takes center stage, with significant traffic declines aligning with late-night traditions like the Midnight Toast (in Argentina, the late-night feast is quite popular) and Misa de Gallo (Midnight Mass). For example:
Colombia: -48% at 21:45
Argentina: -58% at 22:00; -67% at midnight
Chile: -64% at 22:45
Mexico: -50% at 21:45
Brazil: -22% at 21:45
These late-night traffic dips highlight the region’s emphasis on midnight celebrations, family feasts, and religious observances.
Asia Pacific
Asian locations influenced by Western traditions, such as the Philippines and Hong Kong, experience moderate Christmas dips but shift focus to New Year’s celebrations — more on NYE below.
In the Southern Hemisphere, Australia and New Zealand experience their steepest traffic drops during Christmas lunch, with Australia seeing a 43% decrease at 13:45 and New Zealand recording a 42% decline.
Middle East and Africa
In regions less influenced by Christmas, holiday traffic patterns vary significantly. For example, Nigeria sees a 26% drop at lunchtime on Christmas Day, while South Africa records a 37% decline at 14:15, reflecting offline family gatherings.
In predominantly non-Christian countries like Egypt and Saudi Arabia, December 24-25 does not show significant dips; instead, other cultural holidays drive offline moments. For example, as we’ve noted, Israel experienced up to an 8% drop in 2023 during Hanukkah, particularly in the first four days after December 7. In previous blog posts, we have shown how events like Ramadan clearly impact Internet traffic in countries with large Muslim populations. One example from our Year in Review 2024 highlights Indonesia and the United Arab Emirates, where traffic dropped during Eid al-Fitr, the festival marking the end of Ramadan (April 9-10, 2024).
The Boxing Day revival
Boxing Day on December 26 marks a significant digital rebound in countries like the UK, Canada, Australia (where there is a higher increase from the previous week, with daily traffic growing 9%), and New Zealand, as people return online after the Christmas break. Traditionally associated with charitable activities, family gatherings, and shopping, the day sees traffic spikes across these regions:
Location
December 26 increase in daily traffic
Higher traffic increase on December 26
Australia
+9%
December 26; 09:30: +20%
United Kingdom
+2%
December 26; 13:00: +16%
Canada
+1%
December 26, 08:15: +19%
Here is the list of locations that saw a clear drop in traffic on Christmas Eve or Christmas Day morning or lunch. We selected the time (morning or lunch) with the bigger drop compared to the previous week for further analysis. The list is ordered by the Christmas Eve dinner drop. Countries like Russia (where Orthodox Christians celebrate Christmas later, on January 7), Japan, China, Indonesia, Turkey, Israel, Thailand, Egypt, Singapore, Vietnam, and Bangladesh showed no impact during Christmas Eve dinner or Christmas Day morning or lunch.
Location
Christmas Eve Dinner Drop
Christmas Day Morning/Lunch Drop
Spain
-70% at 21:45
-51% at 08:00 (morning)
Portugal
-70% at 20:30
-43% at 13:45 (lunch)
Denmark
-68% at 19:45
-43% at 06:15 (morning)
Chile
-64% at 22:45; (-65% at 00:00, December 25)
-49% at 09:00 (morning)
Norway
-63% at 18:45
-50% at 06:45 (morning)
Czech Republic
-60% at 18:15
-43% at 06:30 (morning)
Poland
-59% at 17:15
-51% at 07:15 (morning)
Argentina
-58% at 22:00 (-67% at 00:00, December 25)
-52% at 09:00 (morning)
Italy
-55% at 21:15
-44% at 07:00 (morning)
France
-55% at 20:45
-44% at 07:15 (morning)
Mexico
-50% at 21:45
-38% at 08:15 (morning)
Belgium
-50% at 20:15
-46% at 07:15 (morning)
Switzerland
-50% at 19:45
-46% at 06:30 (morning)
Austria
-50% at 19:15
-42% at 06:15 (morning)
Nigeria
-49% at 18:00
-26% at 12:30 (lunch)
Colombia
-48% at 21:45
-49% at 08:00 (morning)
Germany
-47% at 19:15
-36% at 07:15 (morning)
Sweden
-47% at 16:30
-36% at 07:00 (morning)
Finland
-42% at 17:45
-42% at 08:00 (morning)
Ireland
-40% at 18:15
-36% at 15:15 (lunch)
South Africa
-37% at 19:00
-37% at 14:15 (lunch)
Romania
-34% at 20:45
-34% at 06:30 (morning)
United Kingdom
-34% at 18:00
-38% at 14:45 (lunch)
Canada
-32% at 20:30
-31% at 09:30 (morning)
Netherlands
-30% at 20:45
-35% at 06:45 (morning)
United States
-29% at 20:15
-32% at 09:15 (morning)
Australia
-23% at 20:45
-43% at 13:45 (lunch)
New Zealand
-23% at 18:30
-42% at 13:15 (lunch)
Brazil
-22% at 21:45
-42% at 08:00 (morning)
Philippines
-22% at 21:30
-29% at 06:45 (morning)
New Year’s Eve: A planetary offline moment
Midnight, December 31 is a shared offline moment worldwide, as people step away from their screens to celebrate. To provide a more accurate assessment of New Year’s Eve’s impact, we compare traffic at 00:00 on January 1 with 00:00 on December 18, avoiding distortions caused by Christmas-related patterns. This approach highlights the distinct drop in Internet activity due to New Year’s celebrations.
Across Europe, countries like Portugal (-60%) and Romania (-60%) see dramatic traffic drops, reflecting widespread offline gatherings. Spain (-56%) and Germany (-49%) also experience steep declines, emphasizing the importance of this tradition across the region. Even Northern Europe mirrors this trend, with Denmark (-41%), Norway (-39%), and Sweden (-29%) showing significant dips.
In the Americas, this offline moment is particularly pronounced in Latin America, where family and communal gatherings dominate. Argentina (-66%) and Chile (-74%) lead the region, with Brazil (-46%) and Colombia (-44%) following closely. In North America, the impact is less dramatic due to time zone variations — in this case, with millions of people spread out in distinct time zones. Canada records a 14% drop, and the United States shows a modest 12% decline compared to December 18.
In Asia and the Pacific, New Year’s Eve celebrations heavily influence Internet trends. Thailand saw a 31% drop, Indonesia 23%, and Japan 16%, also reflecting this region’s focus on communal gatherings and celebrations. Australia (-21%) and New Zealand (-11%), among the first countries to welcome the New Year, also show noticeable declines as midnight festivities take center stage.
In the Middle East and Africa, Turkey (-23%), South Africa (-32%), and Nigeria (-15%) exhibit significant offline engagement at midnight. Israel records a smaller but notable 6% dip before midnight, reflecting localized variations in celebration styles.
Of course, this offline intermission doesn’t last long. After a few hours, people return to their devices. France sees a 37% surge at 3:15 on January 1, while Turkey experiences a 36% upswing in the early hours.
Next, we present the list of locations with clear drops in traffic at midnight on New Year’s Eve, compared to December 18, ordered by percentage of drop.
Locations
January 1, 00:00 drop compared to December 18
Locations
January 1, 00:00 drop compared to December 18
Chile
-74%
Thailand
-31%
Argentina
-66%
Italy
-30%
Romania
-60%
Sweden
-29%
Portugal
-60%
Vietnam
-27%
Spain
-56%
United Kingdom
-25%
Germany
-49%
Ukraine
-25%
Brazil
-46%
Indonesia
-23%
Mexico
-44%
Turkey
-23%
Colombia
-44%
Australia
-21%
Philippines
-43%
Hong Kong
-21%
Netherlands
-42%
Ireland
-19%
Poland
-41%
France
-17%
Denmark
-41%
Japan
-16%
Austria
-40%
South Korea
-16%
Switzerland
-39%
Nigeria
-15%
Norway
-39%
Canada
-14%
Czech Republic
-33%
Finland
-14%
Russia
-32%
Singapore
-13%
Belgium
-32%
United States
-12%
South Africa
-32%
China
-12%
Conclusion: A mosaic of traditions and digital habits
What emerges from these patterns is a rich tapestry of cultural habits. While Christmas Eve and Day are central offline moments in Europe and the Americas, other regions mark their quiet days on different dates, shaped by unique holidays and customs. The insights from 50 countries and regions confirm how cultural traditions guide when people step away from screens.
As the Gregorian calendar year comes to a close, the universal appeal of stepping offline becomes clear. Whether raising glasses at the stroke of midnight, exchanging greetings, or lighting candles for festivals like Hanukkah, these moments remind us that while the Internet connects billions, cultural rhythms still shape our relationship with technology. Whether feasting with loved ones or counting down to a new year, humans everywhere find reasons to unplug — if only for a moment.
]]>Grinch Bots Strike Again: Defending Your Holidays From Cyber Threatshttps://blog.imc.re/RSSBOX/rss/c3d79809.html2024-12-23T14:01:00.000Z2024-12-23T14:01:00.000Z
Grinch Bots are still stealing Christmas
Back in 2021, we covered the antics of Grinch Bots and how the combination of proposed regulation and technology could prevent these malicious programs from stealing holiday cheer.
Fast-forward to 2024 — the Stop Grinch Bots Act of 2021 has not passed, and bots are more active and powerful than ever, leaving businesses to fend off increasingly sophisticated attacks on their own. During Black Friday 2024, Cloudflare observed:
29% of all traffic on Black Friday was Grinch Bots. Humans still accounted for the majority of all traffic, but bot traffic was up 4x from three years ago in absolute terms.
1% of traffic on Black Friday came from AI bots. The majority of it came from Claude, Meta, and Amazon. 71% of this traffic was given the green light to access the content requested.
63% of login attempts across our network on Black Friday were from bots. While this number is high, it was down a few percentage points compared to a month prior, indicating that more humans accessed their accounts and holiday deals.
Human logins on e-commerce sites increased 7-8% compared to the previous month.
These days, holiday shopping doesn’t start on Black Friday and stop on Cyber Monday. Instead, it stretches through Cyber Week and beyond, including flash sales, pre-orders, and various other promotions. While this provides consumers more opportunities to shop, it also creates more openings for Grinch Bots to wreak havoc.
Black Friday - Cyber Monday by the numbers
Black Friday and Cyber Monday in 2024 brought record-breaking shopping — and grinching. In addition to looking across our entire network, we also analyzed traffic patterns specifically on a cohort of e-commerce sites.
Legitimate shoppers flocked to e-commerce sites, with requests reaching an astounding 405 billion on Black Friday, accounting for 81% of the day’s total traffic to e-commerce sites. Retailers reaped the rewards of their deals and advertising, seeing a 50% surge in shoppers week-over-week and a 61% increase compared to the previous month.
Unfortunately, Grinch Bots were equally active. Total e-commerce bot activity surged to 103 billion requests, representing up to 19% of all traffic to e-commerce sites. Nearly one in every five requests to an online store was not a real customer. That’s a lot of resources to waste on bogus traffic. Cyber Week was a battleground, with bots hoarding inventory, exploiting deals, and disrupting genuine shopping experiences.
The upside, if there is one, is that there was more human activity on e-commerce sites (81%) than observed on our network more broadly (71%).
The Grinch Bot’s Modus Operandi
Cloudflare saw 4x more bot requests than what we observed in 2021. Being able to observe and score all this traffic at scale means we can help customers keep the grinches away. We also got to see patterns that help us better identify the concentration of these attacks:
19% of traffic on e-commerce sites was Grinch Bots
1% of traffic to e-commerce sites was from AI Bots.
63% of login attempt requests across our network were from bots
22% of bot activity originated from residential proxy networks
What are all of these bots up to?
AI bots
This year marked a breakthrough for AI-driven bots, agents, and models, with their impact spilling into Black Friday. AI bots went from zero to one, now making up 1% of all bot traffic on e-commerce sites.
AI-driven bots generated 29 billion requests on Black Friday alone, with Meta-external, Claudebot, and Amazonbot leading the pack. Based on their owners, these bots are meant to crawl to augment training data sets for Llama, Claude, and Alexa respectively.
We looked at e-commerce sites specifically to find out if these bots were treating all content equally. While Meta-External and Amazonbot were still in the Top 3 of AI bots reaching e-commerce sites, Bytedance’s Bytespider crawled the most shopping sites.
Account Takeover (ATO) bots
In addition to scraping, crawling, and shopping, bots also targeted customer accounts on Black Friday. We saw 14.1 billion requests from bots to /login endpoints, accounting for 63% of that day’s login attempts.
While this number seems high, intuitively it makes sense, given that humans don’t log in to accounts every day, but bots definitely try to crack accounts every day. Interestingly, while humans only accounted for 36% of traffic to login pages on Black Friday, this number was up 7-8% compared to the prior month. This suggests that more shoppers logged in to capitalize on deals and discounts on Black Friday than in preceding weeks. Human logins peaked at around 40% of all traffic to login sites on the Monday before Thanksgiving, and again on Cyber Monday.
Separately, we also saw a 37% increase in leaked passwords used in login requests compared to the prior month. During Birthday Week, we shared how 65% of internet users are at risk of ATO due to re-use of leaked passwords. This surge, coinciding with heightened human and bot traffic, underscores a troubling pattern: both humans and bots continue to depend on common and compromised passwords, amplifying security risks.
Proxy bots: Regardless of whether they’re crawling your content or hoarding your wares, 22% of bot traffic originated from residential proxy networks. This obfuscation makes these requests look like legitimate customers browsing from their homes rather than large cloud networks. The large pool of IP addresses and the diversity of networks poses a challenge to traditional bot defense mechanisms that rely on IP reputation and rate limiting.
Moreover, the diversity of IP addresses enables the attackers to rotate through them indefinitely. This shrinks the window of opportunity for bot detection systems to effectively detect and stop the attacks. The use of residential proxies is a trend we have been tracking for months now and Black Friday traffic was within the range we’ve seen throughout this year.
If you’re using Cloudflare’s Bot Management, your site is already protected from these bots since we update our bot score based on these types of network fingerprints. In May 2024, we introduced our latest model optimized for detecting residential proxies. Early results show promising declines in this type of activity, indicating that bot operators may be reducing their reliance on residential proxies.
The Christmas “Yule” log: how customers can protect themselves
35% of all traffic on Black Friday was Grinch Bots. To keep Grinch Bots at bay, businesses need year-round bot protection and proactive strategies tailored to the unique challenges of holiday shopping.
(2) Monitor potential Account Takeover (ATO) attacks: Bots often test stolen credentials in the months leading up to Cyber Week to refine their strategies. Re-use of stolen credentials makes businesses even more vulnerable. Our account abuse detections help customers monitor login paths for leaked credentials and traffic anomalies.
Check out more examples of related rules you can create.
(3) Rate limit account and purchase paths: Apply rate-limiting best practices on critical application paths. These include limiting new account access/creation from previously seen IP addresses, and leveraging other network fingerprints, to help prevent promo code abuse and inventory hoarding, as well as identifying account takeover attempts through the application of detection IDs and leaked credential checks.
(4) Block AI bots abusing shopping features to maintain fair access for human users. If you’re using Cloudflare, you can quickly block all AI bots by enabling our automatic AI bot blocking feature.
What to expect in 2025?
Over the next year, e-commerce sites should expect to see more humans shopping for longer periods. As sale periods lengthen (like they did in 2024) we expect more peaks in human activity on e-commerce sites across November and December. This is great for consumers and great for merchants.
More AI bots and agents will be integrated into e-commerce journeys in 2025. AI bots will not only be crawling sites for training data, but will also integrate into the shopping experience. AI bots did not exist in 2021, but now make up 1% of all bot traffic. This is only the tip of the iceberg and their growth will explode in the next year. We expect this to pose new risks as bots mimic and act on behalf of humans.
More sophisticated automation through network, device, and cookie cycling will also become a bigger threat. Bot operators will continue to employ advanced evasion tactics like rotating devices, IP addresses, and cookies to bypass detection.
Grinch Bots are evolving, and regulation may be slowing, but businesses don’t have to face them alone. We remain resolute in our mission to help build a better Internet ... and holiday shopping experience.
Even though the holiday season is closing out soon, bots are never on vacation. It’s never too late or too early to start protecting your customers and your business from grinches that work all year round.
Wishing you all happy holidays and a bot-free new year!
]]>Global Elections in 2024: Internet Traffic and Cyber Threat Trendshttps://blog.imc.re/RSSBOX/rss/ccded7a6.html2024-12-23T14:00:00.000Z2024-12-23T14:00:00.000Z
Elections define the course of democracies (even as there are several types of democracies), and 2024 was a landmark year, with over 60 countries — plus the European Union — holding national elections, impacting half the world’s population. As highlighted in Pew Research’s global elections report, this was a year of “political disruption,” where the Internet was a relevant stage for both democratic engagement and cyber threats.
At Cloudflare, with our presence in over 330 cities and 120 countries and interconnection with 12,500 networks, we’ve witnessed firsthand the digital impact of these elections. From monitoring Internet traffic patterns to mitigating cyberattacks, we’ve observed trends that reveal how elections increasingly play out online. As detailed in our just-published Cloudflare Impact report, we’ve also worked to protect media outlets, political campaigns, and help elections worldwide.
Here’s the map of countries with national elections that took place in 2024, from our elections report.
In terms of Internet patterns, we’ve observed how cyber activity in 2024 continues to intersect with real-world events. Online attacks are clearly a significant part of elections, even when unsuccessful in disrupting candidates or election-related websites due to strong protections. Additionally, Internet traffic patterns often vary on election day depending on the country, and government-directed Internet shutdowns continue, including ones related to elections. Email activity is also influenced, especially for more popular candidates in “polarized battles.”
Let’s start our review with attacks.
Rising threats: political and election-related cyberattacks in 2024
During 2024, elections saw a rise in DDoS attacks targeting political campaigns, parties, and election infrastructure.
In the United States, over 6 billion malicious requests were blocked between November 1-6. A set of DDoS attacks leading up to Election Day on November 5 targeted one of the campaigns with multiple days of attacks, peaking at 700,000 requests per second and sustaining 8 Gbps during major strikes. Key attack tactics included cache-busting, geodiverse patterns, and randomized user agents.
State and local websites also faced increased threats, with 290 million malicious requests blocked since September under Cloudflare’s Athenian Project. Compared to 2020, attacks in 2024 were far more intense, underscoring the growing need for robust cybersecurity to protect elections from disruption.
In France, DDoS attacks plagued multiple political parties, with peaks reaching 96,000 requests per second (rps) on election day, July 7. Additional details are available in our related blog post.
In the United Kingdom, DDoS attacks targeted political parties, with the most severe incident affecting a campaign website, reaching 156,000 rps shortly after the results were announced on election day. Additional details are available in our related blog post.
During the European parliamentary elections in early June, cyberattacks targeted several political websites around election days. Notably, a significant DDoS attack focused on two politically-related websites in the Netherlands on June 5–6 (with June 6 being election day), peaking at 73,000 rps.
In Romania, the weeks leading up to the election cycle culminating in the December 1 parliamentary elections saw DDoS attacks targeting political party websites and news organizations.
In South Africa, where the general election took place on May 29, there was a relevant DDoS attack in the weeks leading up to the election, targeting a major news site within the country for several days, with a peak on May 7 of 54,000 requests per second.
In Portugal, several DDoS attacks targeted political party websites on election day, March 10, particularly after polling stations closed. One political party’s websites experienced a peak of 69,000 rps on May 11 at 00:50 UTC.
In Taiwan, a local fact-checking website faced a DDoS attack three days before the election, on January 10.
In Japan, a DDoS attack targeted a website used to report scams and misinformation a week before the October 27 election.
While some of these rates may seem small to Cloudflare, they can be devastating for websites not well-protected against such high levels of traffic. DDoS attacks not only overwhelm systems but also serve, if successful, as a distraction for IT teams while attackers attempt other types of breaches.
Election-related Internet shutdowns
Several times in 2024, election-related Internet shutdowns were imposed by authorities for various reasons, such as in the Comoros and Pakistan.
Comoros, a small archipelago country in Southeastern Africa with a population of less than 1 million, held presidential elections on January 14, which led to protests against the re-election of President Azali Assoumani. Authorities shut down the Internet on January 17, causing a 50% drop in traffic compared to the previous week, lasting for two days.
Pakistan’s general election day on February 8 was marked by an Internet shutdown targeting mobile networks. The outage began around 02:00 UTC, reducing Internet traffic by 50% compared to the previous week. Traffic only began recovering after 15:00, highlighting the severe impact of government-initiated shutdowns on Internet connectivity.
In Mauritius, an island nation in the Indian Ocean with under 2 million residents, the government suspended access to social media platforms from November 1 to November 11 ahead of the November 10 parliamentary elections.
Other election-related Internet traffic trends
Election-day Internet traffic patterns often reflect a country’s dominant device usage, with mobile-first nations like Indonesia, Mozambique, and Ghana experiencing noticeable traffic drops after polling stations closed. While mobile-friendly countries generally see steady or higher weekend traffic compared to desktop-focused regions like Europe and the Americas, no consistent trend emerged linking device preference to overall election-day traffic increases or decreases.
Here’s a world map from our Year in Review 2024 showing countries where mobile (purple) or desktop (green) dominates Internet traffic.
Now, let’s explore a selection of relevant elections with Internet traffic impacts, ordered by election dates:
Taiwan (January 13)Taiwan’s presidential election saw traffic drop slightly during polling hours, especially in the morning with an 8% drop. Traffic returned to usual levels after 17:00 local time. Post-election, traffic rose by 5% the next morning compared to the previous week.
Finland (January 28)On January 28, Finland held its presidential election. Internet traffic dropped by 24% at 11:00 local time, coinciding with higher voter turnout in the morning. A second noticeable drop of 13% occurred at 20:00 when polling stations closed and TV stations broadcast initial projections, though traffic was slightly higher than usual afterward.
Indonesia (February 14) Indonesia held its general election on February 14. With over 200 million voters spread across 17,000 islands, it likely had the highest number of voters on a single day, unlike India’s multi-week election. During polling hours (08:00 to 13:00 local time), Internet traffic dropped by up to 15%. Traffic remained lower than the previous week for the rest of the day, with drops ranging from 8% to 16% throughout the night. Mobile device usage surged to 77%, the highest of the year, reflecting Indonesia’s mobile-first Internet culture. Traffic recovered the next morning, surpassing the previous week’s levels.
Portugal (March 10)Portugal’s parliamentary election on March 10 saw a sharp 16% traffic drop at 20:00 local time when TV stations began broadcasting projections. Traffic picked up after that and remained stable during the day.
Russia (March 17)Russia’s presidential election showed steady Internet traffic throughout the day but experienced a 7% decrease after polls closed as results and reactions were broadcast on TV. Unlike other countries, where post-election traffic surges are common, Russia’s pattern reflects the strong influence of broadcast media on election coverage.
South Korea (April 10)South Korea held legislative elections on April 10. Traffic was higher than usual before 05:00 local time but dropped 14% by 07:15 after polling stations opened at 06:00. By 11:45, traffic had rebounded above typical levels. After polling stations closed at 18:00, traffic dropped again, with a 7% decline compared to the previous week.
India (April 19–June 1) - related blog postIndia’s seven-phase general election saw significant Internet traffic fluctuations. May 7 recorded the largest nationwide traffic dip of 6%, with populous states like Uttar Pradesh seeing a 9% drop and Maharashtra experiencing a 17% decline. On the final election day (June 1), mobile device usage peaked at 68%, the highest of the year. These patterns underscore India’s mobile-first Internet habits and its diverse election timelines.
North Macedonia (April 24 & May 8)North Macedonia’s two-round presidential election featured a 56% traffic increase after 11:00 local time on May 8, sustained throughout the day. Similar, albeit smaller, trends were observed during the first round on April 24.
Panama (May 5)On May 5, Panama’s presidential and parliamentary election day, Internet traffic dropped significantly while voting stations were open, with a 23% decrease in the afternoon and 25% lower traffic at 21:30 local time as results were announced. Traffic picked up after that.
South Africa (May 29) - related blog postOn May 29, South Africa’s general election saw Internet traffic decrease by 16% at 05:45 and remain lower throughout polling hours. Traffic surged by 25% the night before the election, peaking at midnight. Post-election, traffic increased by up to 12% early on May 30, highlighting the transition from offline to online engagement.
Mexico (June 2) - related blog postMexico’s general election on June 2 saw a 3% daily traffic drop, with hourly dips of up to 11% during polling hours (08:00–20:00 local time). Traffic surged by 14% at 01:30 the following day as results were announced, peaking at 8% above the previous week by 22:00 local time.
Iceland (June 1)Iceland’s presidential election on June 1 saw minor Internet traffic drops, including a 12% dip between 14:00 and 16:00 local time, but traffic increased at night by as much as 11% at 20:00. The day after, traffic rose by 26% compared to the previous week. Iceland elected Halla Tómasdóttir as its second female president.
European Union (June 6–9) - related blog postThe 2024 European Parliament elections showed notable Internet traffic shifts and cybersecurity challenges. The Czech Republic and Slovakia experienced traffic drops of over 10%, while Finland and Ireland saw moderate declines. Key speeches, such as Belgian Prime Minister Alexander De Croo’s resignation and French President Macron’s snap election announcement, also caused traffic fluctuations.
Source: Cloudflare; created with Datawrapper
Iran (June 28)Iran’s presidential election saw significant traffic fluctuations, with traffic falling by 16% after 17:30 local time. Extended polling hours (including at night) led to continued drops, falling to 24% lower by 22:30. After midnight, traffic rebounded, showing a 13% increase compared to the previous week.
France (June 30 & July 7) - related blog postFrance’s legislative elections brought significant Internet and cybersecurity activity. On July 7, Internet traffic dropped 16% at 20:00 local time as polling stations closed and TV broadcasts announced results. Mobile device usage surged to 58%, and DNS traffic to news outlets spiked by 250% during the first round and by 244% on runoff day, reflecting heightened public interest.
United Kingdom (July 4) - related blog postThe UK’s general election on July 4 saw the Labour Party win a majority after 14 years of Conservative rule. Internet traffic declined slightly during voting hours, with a 2% drop at noon, before surging in the evening as results were announced. Northern Ireland experienced the sharpest traffic drop (10%), compared to 6% in Scotland and 5% in Wales. DNS traffic to election-related domains peaked with increases of 600% at 22:00 and 671% at 04:00 the following day.
Sri Lanka (September 21)Sri Lanka’s presidential election caused a 9% morning traffic dip and an 18% post-election surge after polls closed. Results triggered a 109% traffic increase at 03:00 local time on September 22.
Tunisia (October 6)Tunisia’s presidential election saw a 15% traffic dip at 17:00, followed by a 13% decline at 19:30 when results started arriving. The steady traffic decrease highlights the evening focus on offline engagement and result tracking.
Mozambique (October 9)Mozambique’s election drove an Internet traffic drop throughout the day, falling as much as 51% by 20:30 local time, and continuing lower than usual after that. A post-election surge of 16% occurred at 01:30. The election, held on a public holiday, resulted in a 31% daily traffic drop compared to the previous week.
Georgia (October 26)When Georgia held its parliamentary election on October 26, Internet traffic was 11% higher than the previous week, peaking at 67% above normal around 23:00 when results were announced. Unlike other countries, traffic only dipped slightly (2%) in the afternoon during polling hours.
Japan (October 27)Japan’s House of Representatives election saw Internet traffic decrease by 4% at 20:00 after polling stations closed, but it rose later in the evening.
Botswana (October 30)A traffic drop was observed throughout the day of Botswana’s general election, with a 42% decrease around 21:30 local time compared to the previous week.
United States (November 5) - related blog postThe US elections saw a 15% spike in Internet traffic, particularly after polls closed, with the Midwest leading. There were also specific spikes related to key moments during election night, as the next chart shows:
DNS traffic surged by 756% to polling services and 325% to news sites. As highlighted in our recent Internet Services Year in Review blog post, the US election also boosted DNS traffic and ranking positions for CNN, Fox News, and The New York Times, underscoring the Internet’s critical role during major political events.
Ghana (December 7)Ghana’s general election caused mid-morning traffic drops of 11%, followed by declines of 13% and 14% after polling stations closed at 17:00. These patterns indicate offline focus during results announcements.
Romania (December 1)Romania’s parliamentary election showed minimal traffic fluctuations during the day, though its November 24 presidential election remains disputed.
Email perspectives on the US presidential election
From a cybersecurity perspective, trending events, topics, and individuals often attract more emails, including malicious, phishing, and spam messages. In our analysis earlier this year, we focused on the US presidential elections and the two major party candidates.
From June 1 to November 5, 2024, Cloudflare processed over 19 million emails mentioning "Donald Trump" or "Kamala Harris," with Trump appearing more frequently and in higher rates of spam (12%) and malicious emails (1.3%) compared to Harris (0.6% spam, 0.2% malicious). Nearly half were sent after September, with a surge in the final 10 campaign days.
Conclusion: the election cycle doesn’t stop
As a global election year, 2024 underscored how deeply the Internet is woven into the democratic process, serving both as a tool for engagement and a target for disruption. From relevant DDoS attacks to government-imposed Internet shutdowns, the challenges faced during these elections reflect a growing need for robust cybersecurity measures to safeguard critical infrastructure and ensure free, fair electoral processes.
In this context, Germany has announced an anticipated federal election for February 23, 2025, following the collapse of its governing coalition during the 2024 government crisis. This snap election joins others in France and the UK, reflecting a growing trend of political instability requiring urgent electoral responses.
Looking ahead, the increasing frequency and complexity of cyber threats, such as DDoS attacks on campaigns and election infrastructure, demand proactive defenses. Shutdowns like those in Pakistan and Comoros, along with surges in phishing and misinformation, highlight the need for closer collaboration between governments, technology providers, and civil society to safeguard democracy in the digital era.
If you want to follow more trends and insights about the Internet and elections in particular, you can check Cloudflare Radar, and more specifically our new 2024 Elections Insights report.
]]>Hi Claude, Build an MCP Server on Cloudflare Workershttps://blog.imc.re/RSSBOX/rss/9920173b.html2024-12-20T14:50:00.000Z2024-12-20T14:50:00.000Z
In late November 2024, Anthropic announced a new way to interact with AI, called Model Context Protocol (MCP). Today, we’re excited to show you how to use MCP in combination with Cloudflare to extend the capabilities of Claude to build applications, generate images and more. You’ll learn how to build an MCP server on Cloudflare to make any service accessible through an AI assistant like Claude with just a few lines of code using Cloudflare Workers.
A quick primer on the Model Context Protocol (MCP)
MCP is an open standard that provides a universal way for LLMs to interact with services and applications. As the introduction on the MCP website puts it,
“Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools.”
From an architectural perspective, MCP is comprised of several components:
MCP hosts: Programs or tools (like Claude) where AI models operate and interact with different services
MCP clients: Client within an AI assistant that initiates requests and communicates with MCP servers to perform tasks or access resources
MCP servers: Lightweight programs that each expose the capabilities of a service
Local data sources: Files, databases, and services on your computer that MCP servers can securely access
Remote services: External Internet-connected systems that MCP servers can connect to through APIs
Imagine you ask Claude to send a message in a Slack channel. Before Claude can do this, Slack must communicate which tools are available. It does this by defining tools — such as “list channels”, “post messages”, and “reply to thread” — in the MCP server. Once the MCP client knows what tools it should invoke, it can complete the task. All you have to do is tell it what you need, and it will get it done.
Allowing AI to not just generate, but deploy applications for you
What makes MCP so powerful? As a quick example, by combining it with a platform like Cloudflare Workers, it allows Claude users to deploy a Cloudflare Worker in just one sentence, resulting in a site like this:
But that’s just one example. Today, we’re excited to show you how you can build and deploy your own MCP server to allow your users to interact with your application directly from an LLM like Claude, and how you can do that just by writing a Cloudflare Worker.
Simplifying your MCP Server deployment with workers-mcp
The new workers-mcp tooling handles the translation between your code and the MCP standard, so that you don’t have to do the maintenance work to get it set up.
Once you create your Worker and install the MCP tooling, you’ll get a worker-mcp template set up for you. This boilerplate removes the overhead of configuring the MCP server yourself:
import { WorkerEntrypoint } from 'cloudflare:workers'import { ProxyToSelf } from 'workers-mcp'export default class MyWorker extends WorkerEntrypoint<Env> { /** * A warm, friendly greeting from your new Workers MCP server. * @param name {string} the name of the person we are greeting. * @return {string} the contents of our greeting. */ sayHello(name: string) { return `Hello from an MCP Worker, ${name}!` } /** * @ignore **/ async fetch(request: Request): Promise<Response> { return new ProxyToSelf(this).fetch(request) }}
Let’s unpack what’s happening here. This provides a direct link to MCP. The ProxyToSelf logic ensures that your Worker is wired up to respond as an MCP server, without any complex routing or schema definitions.
It also provides tool definition with JSDoc. You’ll notice that the `sayHello` method is annotated with JSDoc comments describing what it does, what arguments it takes, and what it returns. These comments aren’t just for human readers, but they’re also used to generate documentation that your AI assistant (Claude) can understand.
Adding image generation to Claude
When you build an MCP server using Workers, adding custom functionality to an LLM is easy. Instead of setting up the server infrastructure, defining request schemas, all you have to do is write the code. Above, all we did was generate a “hello world”, but now let’s power up Claude to generate an image, using Workers AI:
import { WorkerEntrypoint } from 'cloudflare:workers'import { ProxyToSelf } from 'workers-mcp'
export default class ClaudeImagegen extends WorkerEntrypoint<Env> { /**
Generate an image using the flux-1-schnell model.
@param prompt {string} A text description of the image you want to generate.
@param steps {number} The number of diffusion steps; higher values can improve quality but take longer.
Once you update the code and redeploy the Worker, Claude will now be able to use the new image generation tool. All you have to say is: “Hey! Can you create an image of a lava lamp wall that lives in San Francisco?”
If you’re looking for some inspiration, here are a few examples of what you can build with MCP and Workers:
Let Claude send follow-up emails on your behalf using Email Routing
To build out an MCP server without access to Cloudflare’s tooling, you would have to: initialize an instance of the server, define your APIs by creating explicit schemas for every interaction, handle request routing, ensure that the responses are formatted correctly, write handlers for every action, configure how the server will communicate, and more… As shown above, we do all of this for you.
For reference, an implementation may look something like this:
import { Server } from "@modelcontextprotocol/sdk/server/index.js";import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
const server = new Server({ name: “example-server”, version: “1.0.0” }, { capabilities: { resources: {} } });
server.setRequestHandler(ReadResourceRequestSchema, async (request) => { if (request.params.uri === “file:///example.txt”) { return { contents: [{ uri: “file:///example.txt”, mimeType: “text/plain”, text: “This is the content of the example resource.” }] }; } throw new Error(“Resource not found”); });
const transport = new StdioServerTransport(); await server.connect(transport);
While this works, it requires quite a bit of code just to get started. Not only do you need to be familiar with the MCP protocol, but you need to complete a fair amount of set up work (e.g. defining schemas) for every action. Doing it through Workers removes all these barriers, allowing you to spin up an MCP server without the complexity.
We’re always looking for ways to simplify developer workflows, and we’re excited about this new standard to open up more possibilities for interacting with LLMs, and building agents.
If you’re interested in setting this up, check out this tutorial which walks you through these examples. We’re excited to see what you build. Be sure to share your MCP server creations with us on Discord, X, or Bluesky!
]]>Bring Multimodal Real-Time Interaction to Your AI Applications With Cloudflare Callshttps://blog.imc.re/RSSBOX/rss/ef860f74.html2024-12-20T14:00:00.000Z2024-12-20T14:00:00.000Z
OpenAI announced support for WebRTC in their Realtime API on December 17, 2024. Combining their Realtime API with Cloudflare Calls allows you to build experiences that weren’t possible just a few days earlier.
Previously, interactions with audio and video AIs were largely single-player: only one person could be interacting with the AI unless you were in the same physical room. Now, applications built using Cloudflare Calls and OpenAI’s Realtime API can now support multiple users across the globe simultaneously seeing and interacting with a voice or video AI.
Have your AI join your video calls
Here’s what this means in practice: you can now invite ChatGPT to your next video meeting:
We built this into our Orange Meets demo app to serve as an inspiration for what is possible, but the opportunities are much broader.
In the not-too-distant future, every company could have a 'corporate AI' they invite to their internal meetings that is secure, private and has access to their company data. Imagine this sort of real-time audio and video interactions with your company’s AI:
"Hey ChatGPT, do we have any open Jira tickets about this?"
"Hey Company AI, who are the competitors in the space doing Y?"
"AI, is XYZ a big customer? How much more did they spend with us vs last year?"
There are similar opportunities if your application is built for consumers: broadcasts and global livestreams can become much more interactive. The murder mystery game in the video above is just one example: you could build your own to play live with your friends in different cities.
WebRTC vs. WebSockets
These interactive multimedia experiences are enabled by the industry adoption of WebRTC, which stands for Web Real-time Communication.
Many real-time product experiences have historically used Websockets instead of WebRTC. Websockets operate over a single, persistent TCP connection established between a client and server. This is useful for maintaining a data sync for text-based chat apps or maintaining the state of gameplay in your favorite video game. Cloudflare has extensive support for Websockets across our network as well as in our AI Gateway.
If you were building a chat application prior to WebSockets, you would likely have your client-side app poll the server every n seconds to see if there are new messages to be displayed. WebSockets eliminated this need for polling. Instead, the client and the server establish a persistent, long-running connection to send and receive messages.
However, once you have multiple users across geographies simultaneously interacting with voice and video, small delays in the data sync can become unacceptable product experiences. Imagine building an app that does real-time translation of audio. With WebSockets, you would need to chunk the audio input, so each chunk contains 100–500 milliseconds of audio. That chunking size, along with the head-of-line blocking, becomes the latency floor for your ability to deliver a real-time multimodal experience to your users.
WebRTC solves this problem by having native support for audio and video tracks over UDP-based channels directly between users, eliminating the need for chunking. This lets you stream audio and video data to an AI model from multiple users and receive audio and video data back from the AI model in real-time.
Realtime AI fanout using Cloudflare Calls
Historically, setting up the underlying infrastructure for WebRTC — servers for media routing, TURN relays, global availability — could be challenging.
Cloudflare Calls handles the entirety of this complexity for developers, allowing them to leverage WebRTC without needing to worry about servers, regions, or scaling. Cloudflare Calls works as a single mesh network that automatically connects each user to a server close to them. Calls can connect directly with other WebRTC-powered services such as OpenAI’s, letting you deliver the output with near-zero latency to hundreds or thousands of users.
Privacy and security also come standard: all video and audio traffic that passes through Cloudflare Calls is encrypted by default. In this particular demo, we take it a step further by creating a button that allows you to decide when to allow ChatGPT to listen and interact with the meeting participants, allowing you to be more granular and targeted in your privacy and security posture.
How we connected Cloudflare Calls to OpenAI’s Realtime API
“A Session in Cloudflare Calls correlates directly to a WebRTC PeerConnection. It represents the establishment of a communication channel between a client and the nearest Cloudflare data center, as determined by Cloudflare's anycast routing …
Within a Session, there can be one or more Tracks. … [which] align with the MediaStreamTrack concept, facilitating audio, video, or data transmission.”
To include ChatGPT in our video conferencing demo, we needed to add ChatGPT as a track in an ongoing session. To do this, we connected to the Realtime API in Orange Meets:
This code sets up the bidirectional routing between the human’s session and ChatGPT, which would allow the humans to hear ChatGPT and ChatGPT to hear the humans.
You can review all the code for this demo app on GitHub.
Get started today
Give the Cloudflare Calls + OpenAI Realtime API demo a try for yourself and review how it was built via the source code on GitHub. Then get started today with Cloudflare Calls to bring real-time, interactive AI to your apps and services.
]]>Inside the Research: How GitHub Copilot Impacts the Nature of Work for Open Source Maintainershttps://blog.imc.re/RSSBOX/rss/9f7a102c.html2024-12-20T11:08:03.000Z2024-12-20T11:08:03.000Z
I’m excited to share an interview with two researchers that I’ve had the privilege of collaborating with on a recently released paper studying how open source maintainers adjust their work after they start using GitHub Copilot:
Manuel Hoffmann is a postdoctoral scholar at the Laboratory for Innovation Science at Harvard housed within the Digital, Data, and Design Institute at Harvard Business School. He is also affiliated with Stanford University. His research interests lie in social and behavioral aspects around open source software and artificial intelligence, under the broader theme of innovation and technology management, with the aim of better understanding strategic aspects for large, medium-sized, and entrepreneurial firms.
Sam Boysel is a postdoctoral fellow at the Laboratory for Innovation Science at Harvard. He is an applied microeconomist with research interests at the intersection of digital economics, labor and productivity, industrial organization, and socio-technical networks. Specifically, his work has centered around the private provision of public goods, productivity in open collaboration, and welfare effects within the context of open source software (OSS) ecosystems.
Research Q&A
Kevin: Thanks so much for chatting, Manuel and Sam! So, it seems like the paper’s really making the rounds. Could you give a quick high-level summary for our readers here?
Manuel: Thanks to the great collaboration with you, Sida Peng from Microsoft and Frank Nagle from Harvard Business School, we study the impact of GitHub Copilot on developers and how this generative AI alters the nature of work. We find that when you provide developers in the context of open source software with a generative AI tool that reduces the cost of the core work of coding, developers increase their coding activities and reduce their project management activities. We find our results are strongest in the first year after the introduction but still exist even two years later. The results are driven by developers who are working more autonomously and less collaboratively since they do not have to engage with other humans to solve a problem but they can solve the problem through AI assistance.
GitHub Copilot, on average, increases the share of coding for maintainers who use it. These effects are strongest in the first year, but they still persist over the course of two years.GitHub Copilot, on average, reduces the share of project management for maintainers who use it. These effects are strongest in the first year, but still persist over the course of two years.
Sam: That’s exactly right. We tried to even further understand the nature of work by digging into the paradigm of exploration vs exploitation. Loosely speaking, exploitation is the idea to exert effort towards the most lucrative of the already known options while exploration implies to experiment to find new options with a higher potential return. We tested this idea in the context of GitHub. Developers that had access to GitHub Copilot are engaged in more experimentation and less exploitation, that is, they start new projects with access to AI and have a lower propensity to work on older projects. Additionally, they expose themselves to more languages that they were previously not exposed to and in particular to languages that are valued higher in the labor market. A back-of-the-envelope calculation from purely experimentation among new languages due to GitHub Copilot suggests a value of around half a billion USD within a year.
Kevin: Interesting! Could you provide an overview of the methods you used in your analysis?
Manuel: Would be happy to! We are using a regression discontinuity design in this work, which is as close as you can get to a randomized control trial when purely using pre-existing data, such as the one from GitHub, without introducing randomization by the researcher. Instead, the regression discontinuity design is based on a ranking of developers and a threshold that GitHub uses to determine eligibility for free access to GitHub Copilot’s top maintainer program.
Sam: The main idea of this method is that a developer that is right below the threshold is roughly identical to a developer that is right above the threshold. Stated differently, by chance a developer happened to have a ranking that made them eligible for the generative AI while they could as well not have been eligible. Taken together with the idea that developers neither know the threshold nor the internal ranking from GitHub, we can be certain that the changes that we observe in coding, project management, and the other activities on the platform are only driven by the developers having access to GitHub Copilot and nothing else.
Kevin: Nice! Follow-up question: could you provide an “explain like I’m 5” overview of the methods you used in your analysis?
Manuel: Sure thing, let’s illustrate the problem a bit more. Some people use GitHub Copilot and others don’t. If we just looked at the differences between people who use GitHub Copilot vs. those who don’t, we’d be able to see that certain behaviors and characteristics are associated with GitHub Copilot usage. For example, we might find that people who use GitHub Copilot push more code than those who don’t. Crucially, though, that would be a statement about correlation and not about causation. Often, we want to figure out whether X causes Y, and not just that X is correlated with Y. Going back to the example, if it’s the case that those who use GitHub Copilot push more code than those who don’t, these are a few of the different explanations that might be at play:
Using GitHub Copilot causes people to push more code.
Pushing more code causes people to use GitHub Copilot.
There’s something else that both causes people to use GitHub Copilot and causes them to push more code (for example, being a professional developer).
Because (1), (2), and (3) could each result in data showing a correlation between GitHub Copilot usage and code pushes, just finding a correlation isn’t super interesting. One way you could isolate the cause and effect relationship between GitHub Copilot and code pushes, though, is through a randomized controlled trial (RCT).
In an RCT, we randomly assign people to use GitHub Copilot (the treatment group), while others are forbidden from using GitHub Copilot (the control group). As long as the assignment process is truly random and the users comply with their assignments, any outcome differences between the treatment and control groups can be attributed to GitHub Copilot usage. In other words, you could say that GitHub Copilot caused those effects. However, as anyone in the healthcare field can tell you, large-scale RCTs over long-time periods are often prohibitively expensive, as you’d need to recruit subjects to participate, monitor them to see if they complied with their assignments, and follow up with them over time.
Sam: That’s right. So, instead, wouldn’t it be great if there would be a way to observe developers without running an RCT and still draw valid causal conclusions about GitHub Copilot usage? That’s where the regression discontinuity design (RDD) comes in. The random assignment aspect of an RCT allows us to compare the outcomes of two virtually identical groups. Sometimes, however, randomness already exists in a system, which we can use as a natural experiment. In the case of our paper, this randomness came in the form of GitHub’s internal ranking for determining which open source maintainers were eligible for free access to GitHub Copilot.
Let’s walk through a simplified example. Let’s imagine that there were one million repositories that were ranked on some set of metrics and the rule was that the top 500,000 repositories are eligible for free access to GitHub Copilot. If we compared the #1 ranked repository with the #1,000,000 ranked repository, then we would probably find that those two are quite different from each other. After all, the #1 repository is the best repository on GitHub by this metric while the #1,000,000 repository is a whole 999,999 rankings away from it. There are probably meaningful differences in code quality, documentation quality, project purpose, maintainer quality, etc. between the two repositories, so we would not be able to say that the only reason why there was a difference in outcomes for the maintainers of repository #1 vs. repository #1,000,000 was because of free access to GitHub Copilot.
However, what about repository #499,999 vs. repository #500,001? Those repositories are probably very similar to each other, and it was all down to random chance as to which repository made it over the eligibility threshold and which one did not. As a result, there is a strong argument that any differences in outcomes between those two repositories is solely due to repository #499,999 having free access to GitHub Copilot and repo #500,001 not having free access. Practically, you’ll want to have a larger sample size than just two, so you would compare a narrow set of repositories just above and below the eligibility threshold against each other.
Kevin: Thanks, that’s super helpful. I’d be curious about the limitations of your paper and data that you wished you had for further work. What would the ideal dataset(s) look like for you?
Manuel: Fantastic question! Certainly, no study is perfect and there will be limitations. We are excited to better understand generative AI and how it affects work in the future. As such, one limitation is the availability of information from private repositories. We believe that if we were to have information on private repositories we could test whether there is some more experimentation going on in private projects and that project improvements that were done with generative AI in private may spill over to the public to some degree over time.
Sam: Another limitation of our study is the language-based exercise to provide a value of GitHub Copilot. We show that developers focus on higher value languages that they did not know previously and we extrapolated this estimate to all top developers. However, this estimate is certainly only a partial equilibrium value since developer wages may change over time in a full equilibrium situation if more individuals offer their services for a given language. However, despite the limitation, the value seems to be an underestimate since it does not contain any non-language specific experimentation value and non-experimentation value that is derived from GitHub Copilot.
Kevin: Predictions for the future? Recommendations for policymakers? Recommendations for developers?
Manuel: One simple prediction for the future is that AI incentivizes the activity for which it lowers the cost. However, it is not clear yet which parts will be incentivized through AI tools since they can be applied to many domains. It is likely that there are going to be a multitude of AI tools that incentivize different work activities which will eventually lead to employees, managers at firms, and policy-makers having to consider on which activity they want to put weight. We also would have to think about new recombinations of work activities. Those are difficult to predict. Avi Goldfarb, a prolific professor from the University of Toronto, gave an example of the steam engine with his colleagues. Namely, work was organized in the past around the steam engine as a power source but once electric motors were invented, that was not necessary anymore and structural changes happened. Instead of arranging all of the machinery around a giant steam engine in the center of the factory floor, electricity enabled people to design better arrangements, which led to boosts in productivity. I find this historical narrative quite compelling and can imagine similarly for AI that the greatest power still remains to be unlocked and that it will be unlocked once we know how work processes can be re-organized. Developers can think as well about how their work may change in the future. Importantly, developers can actively shape that future since they are closest to the development of machine learning algorithms and artificial intelligence technologies.
Sam: Adding on to those points, it is not clear when work processes change and whether it will have an inequality reducing or enhancing effect. Many predictions point towards an inequality enhancing effect since the training of large-language models requires substantial computing power which is often only in the hands of a few players. On the other hand, it has been documented that especially lower ability individuals seem to benefit the most from generative AI, at least in the short-term. As such, it’s imperative to understand how the benefits of generative AI are distributed across society. If not, are there equitable, welfare-improving interventions that can correct these imbalances? An encouraging result of our study suggests that generative AI can be especially impactful for relatively lesser skilled workers:
Low ability developers are increasing their coding activity by more than high ability developers. The Max Centrality proxy measures the largest share of commits the developer made across all public repositories where they have made at least one commit. The Achievements proxy counts the cumulative total of “badges” the developer has earned on the GitHub platform. The Followers proxy counts the number of peers who subscribe to notifications for the developer’s activity on GitHub. The Tenure proxy is measured by counting the number of days since the developer created their GitHub account. A developer would be considered to be “high ability” for a proxy if they were above the median for that proxy and “low ability” for a proxy if they were below the median for that proxy.Low ability developers are reducing their project management activity by more than high ability developers.
Sam (continued): Counter to widespread speculation that generative AI will replace many entry level tasks, we find reason to believe that AI can also lower the costs of experimentation and exploration, reduce barriers to entry, and level the playing field in certain segments of the labor market. It would be prudent for policymakers to monitor distributional effects of generative AI, allowing the new technology to deliver equitable benefits where it does so naturally but at the same time intervening in cases where it falls short.
Personal Q&A
Kevin: I’d like to change gears a bit to chat more about your personal stories. Manuel, I know you’ve worked on research analyzing health outcomes with Stanford, diversity in TV stations, and now you’re studying nerds on the internet. Would love to learn about your journey to getting there.
Manuel: Sure! I was actually involved with “nerds on the internet” longer than my vita might suggest. Prior to my studies, I was using open source software, including Linux and Ubuntu, and programming was a hobby for me. I enjoyed the freedom that one had on the personal computer and the internet. During my studies, I discovered economics and business studies as a field of particular interest. Since I was interested in causal inference and welfare from a broader perspective, I learned how to use experimental and quasi-experimental studies to better understand social, medical and technological innovation that are relevant for individuals, businesses, and policy makers. I focused on labor and health during my PhD and afterwards I was able to lean a bit more into health at Stanford University. During my time at Harvard Business School, the pendulum swung back a bit towards labor. As such, I was in the fortunate position—thanks to the study of the exciting field of open source software—to continuously better understand both spaces.
Kevin: Haha, great to hear your interest in open source runs deep! Sam, you also have quite the varied background, analyzing cleantech market conditions and the effects of employment verification policies, and you also seem to have been studying nerds on the internet for the past several years. Could you share a bit about your path?
Sam: I’ve been a computing and open source enthusiast since I got my hands on a copy of “OpenSUSE for Dummies” in middle school. As an undergraduate, I was drawn to the social science aspect of economics and its ability to explain or predict human behavior across a wide range of settings. After bouncing around a number of subfields in graduate school, I got the crazy idea to combine my field of study with my passion and never looked back. Open source is an incredibly data-rich environment with a wealth of research questions interesting to economists. I’ve studied the role of peer effects in driving contribution, modelled the formation of software dependency networks using strategic behavior and risk aversion, and explored how labor market competition shapes open source output.
And thanks, Kevin. I’ll be sure to work “nerds on the internet” into the title of the next paper.
Kevin: Finding a niche that you’re passionate about is such a joy, and I’m curious about how you’ve found living in that niche. What’s the day-to-day like for you both?
Manuel: The day-to-day can vary but as an academic, there are a few tasks that are recurring at a big picture level. Research, teaching and other work. Let’s focus on the research bucket. I am quite busy with working on causal inference papers, refining them but also speaking to audiences to communicate our work. Some of the work is done jointly, some by oneself, so there is a lot of variation, and the great part of being an academic is that one can choose that variation oneself through the projects one selects. Over time one has to juggle many balls. Hence, I am working on finishing prior papers in the space of health; for example, some that you alluded to previously on Television, Health and Happiness and Vaccination at Work, but also in the space of open source software, importantly, to improve the paper on Generative AI and the Nature of Work. We have continuously more ideas to better understand the world we are living and going to live in at the intersection of open source software and generative AI and, as such, it is very valuable to relate to the literature and eventually answer exciting questions around the future of work with real world data. GitHub is a great resource for that.
Sam: As an applied researcher, I use data to answer questions. The questions can come from current events, from conversations with both friends and colleagues, or simply musing on the intricacies of the open source space. Oftentimes the data comes from publicly observed behavior of firms and individuals recorded on the internet. For example, I’ve used static code analysis and version control history to characterize the development of open source codebases over time. I’ve used job postings data to measure firm demand for open source skills. I’ve used the dependency graphs of packaging ecosystems to track the relationships between projects. I can then use my training in economic theory, econometric methodology, and causal inference to rigorously explore the question. The end result is written up in an article, presented to peers, and iteratively improved from feedback.
Kevin: Have things changed since generative AI tooling came along? Have you found generative AI tools to be helpful?
Manuel: Definitely. I use GitHub Copilot when developing experiments and programming in Javascript together with another good colleague, Daniel Stephenson from Virginia Commonwealth University. It is interesting to observe how often Copilot actually makes code suggestions based on context that are correct. As such, it is an incredibly helpful tool. However, the big picture of what our needs are can only be determined by us, as such, in my experience Copilot does seem to speed up the process and leads to avoiding some mistakes conditional on not just blindly following the AI.
Sam: I’ve only recently begun using GitHub Copilot, but it’s had quite an impact on my workflow. Most social sciences researchers are not skilled software engineers. However, they must also write code to hit deadlines. Before generative AI, the delay between problem and solution was usually characterized by many search queries, parsing Q&A forums or documentation, and a significant time cost. Being able to resolve uncertainty within the IDE is incredible for productivity.
Kevin: Advice you might have for folks who are starting out in software engineering or research? What tips might you give to a younger version of yourself, say, from 10 years ago?
Manuel: I will talk a bit at a higher level. Find work, questions or problems that you deeply care about. I would say that is a universal rule to be satisfied, be it in software engineering, research or in any other area. In a way, think about the motto, “You only live once,” as a pretty applicable misnomer. Another universal advice that has high relevance is to not care too much about the things you cannot change but focus on what you can control. Finally, think about getting advice from many different people, then pick and choose. People have different mindsets and ideas. As such, that can be quite helpful.
Sam:
Being able to effectively communicate with both groups (software eng and research) is extremely important.
You’ll do your best work on things you’re truly passionate about.
Premature optimization truly is the root of all evil.
Kevin: Learning resources you might recommend to someone interested in learning more about this space?
Manuel: There are several aspects we have touched upon—Generative AI Research, Coding with Copilot, Causal Inference and Machine Learning, Economics, and Business Studies. As such, here is one link for each topic that I can recommend:
I am sure that there are many other great resources out there, many that can also be found on GitHub.
Sam: If you’re interested to get more into how economists and other social scientists think about open source, I highly recommend the following (reasonably entry-level) articles that have helped shape my research approach.
Kevin: Thank you, Manuel and Sam! We really appreciate you taking the time to share about what you’re working on and your journeys into researching open source.
]]>OpenAI’s Latest O1 Model Now Available in GitHub Copilot and GitHub Modelshttps://blog.imc.re/RSSBOX/rss/9bb226be.html2024-12-20T09:23:42.000Z2024-12-20T09:23:42.000Z
Hey devs! We have some exciting news for ya.
So, backstory first: in case you missed it, OpenAI launched its o1-preview and o1-mini models back in September, optimized for advanced tasks like coding, science, and math. We brought them to our GitHub Copilot Chat and to GitHub Models so you could start playing with them right away, and y’all seemed to love it!
Fast forward to today, and OpenAI has just shipped the release version to o1, an update to o1-preview with improved performance in complex tasks, including a 44% gain in Codeforces’ competitive coding test.
Sounds pretty good, huh? Well, exciting news: We’re bringing o1 to you in Copilot Chat across Copilot Pro, Business, and Enterprise, and including GitHub Models. It’s also available for you to start using right now!
How can I try o1 out in chat?
In Copilot Chat in Visual Studio Code and on GitHub, you can now pick “o1 (Preview)” via the model picker if you have a paid Copilot subscription (this is not available on our newly announced free Copilot tier).
Before, it said “o1-preview (Preview)” and now it just says “o1 (Preview)” because the GitHub model picker is in preview, but not the model itself. We have proven once again that the hardest problem in computer science is naming things, but yo dawg, we no longer have previews on previews. We are a very professional business.
Anyway.
Now you can use Copilot with o1 to explain, debug, refactor, modernize, test… all with the latest and greatest that o1 has to offer!
o1 is included in your paid subscription to GitHub Copilot, allowing up to 10 messages every 12 hours. As a heads up, if you’re a Copilot Business or Copilot Enterprise subscriber, an administrator will have to enable access to o1 models first before you can use it.
Beyond the superpowers that o1 already gives you out of the box, it also pulls context from your workspace and GitHub repositories! It’s a match made in developer heaven.
What about o1 in GitHub Models?
If you haven’t played with GitHub Models yet, you’re in for a treat. It lets you start building AI apps with a playground and an API (and more). Learn more about GitHub Models in our documentation and start experimenting and building with a variety of AI models today on the GitHub Marketplace.
Head here to start with o1 in the playground, plus you can compare it to other models from providers such as Mistral, Cohere, Microsoft, or Meta, and use the code samples you get to start building. Every model is different, and this kind of experimentation is really helpful to get the results you want!
It’s all about developer choice!
Having choices fuels developer creativity! We don’t want you to just write better code, but for you to have the freedom to build, innovate, commit, and push in the way that works best for you.
I hope you’re as excited as I am. GitHub is doubling down on its commitment to give you the most advanced tools available with OpenAI’s new o1 model, and if you keep your eye out, there’s even better things to come.
]]>Enhance Build Security and Reach SLSA Level 3 With GitHub Artifact Attestationshttps://blog.imc.re/RSSBOX/rss/92d6d03c.html2024-12-20T08:59:34.000Z2024-12-20T08:59:34.000Z
The need for software build security is more pressing than ever. High-profile software supply chain attacks like SolarWinds, MOVEit, 3CX, and Applied Materials have revealed just how vulnerable the software build process can be. As attackers exploit weaknesses in the build pipeline to inject their malicious components, traditional security measures—like scanning source code, securing production environments and source control allow lists—are no longer enough. To defend against these sophisticated threats, organizations must treat their build systems with the same level of care and security as their production environments.
These supply chain attacks are particularly dangerous because they can undermine trust in your business itself: if an attacker can infiltrate your build process, they can distribute compromised software to your customers, partners, and end-users. So, how can organizations secure their build processes, and ensure that what they ship is exactly what they intended to build?
The Supply-chain Levels for Software Artifacts (SLSA) framework was developed to address these needs. SLSA provides a comprehensive, step-by-step methodology for building integrity and provenance guarantees into your software supply chain. This might sound complicated, but the good news is that GitHub Artifact Attestations simplify the journey to SLSA Level 3!
In this post, we’ll break down what you need to know about SLSA, how Artifact Attestations work, and how they can boost your GitHub Actions build security to the next level.
Securing your build process: An introduction to SLSA
What is build security?
When we build software, we convert source code into deployable artifacts—whether those are binaries, container images or packaged libraries. This transformation occurs through multiple stages, such as compilation, packaging and testing, each of which could potentially introduce vulnerabilities or malicious modifications.
A properly secured build process can:
Help ensure the integrity of your deployed artifacts by providing a higher level of assurance that the code has not been tampered with during the build process.
Provide transparency into the build process, allowing you to audit the provenance of your deployed artifacts.
Maintain confidentiality by safeguarding sensitive data and secrets used in the build process.
By securing the build process, organizations can ensure that the software reaching end-users is the intended and unaltered version. This makes securing the build process just as important as securing the source code and deployment environments.
Introducing SLSA: A framework for build security
SLSA is a community-driven framework governed by the Open Source Security Foundation (OpenSSF), designed to help organizations systematically secure their software supply chains through a series of progressively stronger controls and best practices.
The framework is organized into four levels, each representing a higher degree of security maturity:
Level 0: No security guarantees
Level 1: Provenance exists for traceability, but minimal tamper resistance
Level 2: Provenance signed by a managed build platform, deterring simple tampering
Level 3: Provenance from a hardened, tamper-resistant build platform, ensuring high security against compromise
Provenance refers to the cryptographic record generated for each artifact, providing an unforgeable paper trail of its build history. This record allows you to trace artifacts back to their origins, allowing for verification of how, when and by whom the artifact was created.
Why SLSA Level 3 matters for build security
Achieving SLSA Level 3 is a critical step in building a secure and trustworthy software supply chain. This level requires organizations to implement rigorous standards for provenance and isolation, ensuring that artifacts are produced in a controlled and verifiable manner. An organization that has achieved SLSA Level 3 is capable of significantly mitigating the most common attack vectors targeting software build pipelines. Here’s a breakdown of the specific requirements for reaching SLSA Level 3:
Provenance generation and availability: A detailed provenance record must be generated for each build, documenting how, when and by whom each artifact was produced. This provenance must be accessible to users for verification.
Managed build system: Builds must take place on ephemeral build systems—short-lived, on-demand environments that are provisioned for each build in order to isolate builds from one another, reducing the risk of cross-contamination and unauthorized access.
Restricted access to signing material: User-defined build steps should not have access to sensitive signing material to authenticate provenance, keeping signing operations separate and secure.
GitHub Artifact Attestations help simplify your journey to SLSA Level 3 by enabling secure, automated build verification within your GitHub Actions workflows. While generating build provenance records is foundational to SLSA Level 1, the key distinction at SLSA Level 3 is the separation of the signature process from the rest of your build job. At Level 3, the signing happens on dedicated infrastructure, separated from the build workflow itself.
The importance of verifying signatures
While signing artifacts is a critical step, it becomes meaningless without verification. Simply having attestations does not provide any security advantages if they are not verified. Verification ensures that the signed artifacts are authentic and have not been tampered with.
The GitHub CLI makes this process easy, allowing you to verify signatures at any stage of your CI/CD pipeline. For example, you can verify Terraform plans before applying them, ensure that Ansible or Salt configurations are authentic before deployment, validate containers before they are deployed to Kubernetes, or use it as part of a GitOps workflow driven by tools like Flux.
GitHub offers several native ways to verify Artifact Attestations:
GitHub CLI: This is the easiest way to verify signatures.
Kubernetes Admission Controller: Use GitHub’s distribution of the admission controller for automated verification in Kubernetes environments.
Offline verification: Download the attestations and verify them offline using the GitHub CLI for added security and flexibility in isolated environments.
By verifying signatures during deployment, you can ensure that what you deploy to production is indeed what you built.
Achieving SLSA Level 3 compliance with GitHub Artifact Attestations
Reaching SLSA Level 3 may seem complex, but GitHub’s Artifact Attestations feature makes it remarkably straightforward. Generating build provenance puts you at SLSA Level 1, and by using GitHub Artifact Attestations on GitHub-hosted runners, you reach SLSA Level 2 by default. From this point, advancing to SLSA Level 3 is a straightforward journey!
The critical difference between SLSA Level 2 and Level 3 lies in using a reusable workflow for provenance generation. This allows you to centrally enforce build security across all projects and enables stronger verification, as you can confirm that a specific reusable workflow was used for signing. With just a few lines of YAML added to your workflow, you can gain build provenance without the burden of managing cryptographic key material or setting up additional infrastructure.
Build provenance made simple
GitHub Artifact Attestations streamline the process of establishing provenance for your builds. By enabling provenance generation directly within GitHub Actions workflows, you ensure that each artifact includes a verifiable record of its build history. This level of transparency is crucial for SLSA Level 3 compliance.
Best of all, you don’t need to worry about the onerous process of handling cryptographic key material. GitHub manages all of the required infrastructure, from running a Sigstore instance to serving as a root signing certificate authority for you.
Check out our earlier blog to learn more about how to set up Artifact Attestations in your workflow.
Secure signing with ephemeral machines
GitHub Actions-hosted runners, executing workflows on ephemeral machines, ensure that each build process occurs in a clean and isolated environment. This model is fundamental for SLSA Level 3, which mandates secure and separate handling of key material used in signing.
When you create a reusable workflow for provenance generation, your organization can use it centrally across all projects. This establishes a consistent, trusted source for provenance records. Additionally, signing occurs on dedicated hardware that is separate from the build machine, ensuring that neither the source code nor the developer triggering the build system can influence or alter the build process. With this level of separation, your workflows inherently meet SLSA Level 3 requirements.
Below is an example of a reusable workflow that can be utilized across the organization to sign artifacts:
name: Sign Artifact
on: workflow_call: inputs: artifact-path: required: true type: string
When you want to use this reusable workflow for signing in any other workflow, you can call it as follows:
name: Sign Artifact Workflow
on: push: branches: - main
jobs: sign: runs-on: ubuntu-latest
steps:- name: Sign Artifact uses: <repository>/.github/workflows/sign-artifact.yml@<version> with: subject-name: "your-artifact.tar.gz" # Replace with actual artifact name subject-digest: "your-artifact-digest" # Replace with SHA-256 digest
This architecture of ephemeral environments and centralized provenance generation guarantees that signing operations are isolated from the build process itself, preventing unauthorized access to the signing process. By ensuring that signing occurs in a dedicated, controlled environment, the risk of compromising the signing workflow can be greatly reduced so that malicious actors can not tamper with the signing action’s code and deviate from the intended process. Additionally, provenance is generated consistently across all builds, providing a unified record of build history for the entire organization.
To verify that an artifact was signed using this reusable workflow, you can use the GitHub CLI with the following command:
This verification process ensures that the artifact was built and signed using the anticipated pipeline, reinforcing the integrity of your software supply chain.
A more secure future
GitHub Artifact Attestations bring the assurance and structure of SLSA Level 3 to your builds without having to manage additional security infrastructure. By simply adding a few lines of YAML and moving the provenance generation into a reusable workflow, you’re well on your way to achieving SLSA Level 3 compliance with ease!
Ready to strengthen your build security and achieve SLSA Level 3?
Start using GitHub Artifact Attestations today or explore our documentation to learn more.
]]>The Role of Email Security in Reducing User Risk Amid Rising Threatshttps://blog.imc.re/RSSBOX/rss/5951e5e0.html2024-12-19T14:00:00.000Z2024-12-19T14:00:00.000Z
Phishing remains one of the most dangerous and persistent cyber threats for individuals and organizations. Modern attacks use a growing arsenal of deceptive techniques that bypass traditional secure email gateways (SEGs) and email authentication measures, targeting organizations, employees, and vendors. From business email compromise (BEC) to QR phishing and account takeovers, these threats are designed to exploit weaknesses across multiple communication channels, including email, Slack, Teams, SMS, and cloud drives.
Phishing remains the most popular attack vector for bad actors looking to gain unauthorized access or extract fraudulent payment, and it is estimated that 90% of all attacks start with a phishing email. However, as companies have shifted to using a multitude of apps to support communication and collaboration, attackers too have evolved their approach. Attackers now engage employees across a combination of channels in an attempt to build trust and pivot targeted users to less-secure apps and devices. Cloudflare is uniquely positioned to address this trend thanks to our integrated Zero Trust services, extensive visibility from protecting approximately 20% of all websites, and signals derived from processing billions of email messages a year.
Cloudflare recognizes that combating phishing requires an integrated approach and a more complete view of user-based risk. That’s why we’ve designed our email security solution to protect organizations before, during, and after message delivery, while also extending protection beyond email into the broader security ecosystem. Phishing is no longer just an email problem — it’s a multi-channel, cross-application threat.
Assessing holistic user risk
When it comes to protecting against user-based threats, Cloudflare employs a platform approach to security. Instead of forcing customers to rely on an array of fragmented tools that create unnecessary complexity and blind spots, we treat email security as part of an overall strategy for assessing and responding to user-related risk. Our email security solution works in tandem with our network solutions so that SOC teams can quickly assert what actions their users are performing outside of email. Given our extensive network visibility, our platform is not limited by API integrations, and can provide SOC teams with the best visibility and protection. This helps SOC teams not only combat phishing, but begin to identify and take action against a wider range of insider threats.
Within a single, unified dashboard, SOC teams can quickly review detailed information regarding the following questions, which we discuss in more detail below:
Who in the organization is being targeted?
Who are the attackers impersonating?
What risky behaviors are my users performing?
Who in the organization is being targeted?
Within the Cloudflare dashboard, SOC teams can view which users are the most targeted. This can help them determine which accounts should be hardened (e.g. MFA enforced), and identify risky users that should be monitored more closely for significant deviations in behavior. One way organizations can use this information is to require high-risk users to connect from a managed device. For instance, if they use Crowdstrike, we can require that these users be on a managed device and force a posture check before letting them access sensitive applications.
SOC teams can also dive into what types of attacks are hitting their users and at what frequency.
Customers can use these insights to adjust various platform policies, effectively blocking malicious content and securing sensitive resources. Above, we can see that attackers are frequently leveraging links to try to compromise users. Based on the link analysis we are seeing in email, SOC teams can use our gateway to block similar attacks, so that when attackers try to use other communication methods (LinkedIn, Teams, Slack, etc.) users will not be able to interact with those links.
SOC teams can also get visibility into impersonation attempts within their email environment. Customers can see which users are being impersonated the most, and can use this information to build policies within our email security solution and broader set of Zero Trust services.
A list of frequently impersonated users can be added to the impersonation registry, which changes the sensitivity of our models to apply more scrutiny on messages coming from those users.
Given our unique position as a domain name registrar, customers can also report lookalike domains to Cloudflare for action to be taken against them. This helps prevent attackers from being able to impersonate our customers and negatively impact their reputation.
Finally, customers can also use our free DMARC management to track who is sending emails on their behalf. This information can be used to update SPF records and get customers to p=quarantine or p=reject so that their brand is more resistant to being spoofed.
What risky behaviors are my users performing?
Cloudflare provides visibility into user actions in several ways.
Within the email security solution, we can track internal messages and alert if we see any malicious or suspicious behaviors. This can be enhanced with our managed service offering, Phishguard, which can alert admins when they see any type of behavior that indicates fraud (like Business Email Compromise), account takeover, or insider threats.
SOC teams can also take advantage of our CASB solution to view the different actions that users have performed. Actions are labeled with different risk levels to let teams know which findings are critical and require remediation.
Customers are also able to view data loss prevention (DLP) violations that users have incurred to see if there is any unauthorized egress of data. We provide the ability to automatically block this egress based on different policies within our platform, making sure there is no exfiltration of sensitive data.
We also enable organizations to put internal applications behind our Access solution. This prevents any users with improper permissions or a high risk level from accessing critical applications. Our dashboard then provides metrics on these logins to see how many failures we observed, so that SOC teams can investigate the user further.
These signals feed into our Unified Risk Score, which can be exported if needed to take automated actions within other platforms.
Increasing SOC productivity
With all of our functionality unified within a single interface and fed by one data lake, we see an increase in SOC productivity because teams no longer have to spend time building rules or flipping between disparate interfaces and workflows.
AI-driven email security
Unlike legacy secure email gateways, our email security solution is driven by predictive AI models which eliminate the need for creating and updating rules. These models are also more effective than reactive measures because they are fed by a massive volume of diverse data from across Cloudflare’s network. This means models are trained on emerging threats earlier and can identify new tactics with a higher accuracy than legacy systems.
Automated isolation
To further reduce the risk posed by users visiting potentially malicious websites, customers can isolate browser sessions using our natively integrated, clientless remote browser that runs on our global network. Within an isolated browsing session, SOC teams can prohibit various behaviors such as copy/paste, upload/download, keyboard inputs, and more. This decreases the risk of users accessing a website and performing an action which could compromise the organization.
Our browser isolation solution also decreases the time SOC teams need to maintain policies. Rather than adding domains and applications one by one, teams can choose to isolate based on content categories. These categories are based on our threat intelligence, and are constantly updated. This means that as new websites emerge, SOC teams do not have to spend the time to chase down and update the proper policy — rather, it is done automatically.
Automated blocking
While some websites might require running in an isolated browser to mitigate the risk of users encountering malicious content, others may need to be fully blocked altogether. Customers can use the same process listed above to block any website that could be risky for users based on tags. However, we allow admins to also provide feedback to users to increase awareness. This can be done via a custom block page that allows SOC teams to communicate with users about their risky behaviors, so that they take actions to curb this behavior in the future and alert their SOC teams to attacks that might be occurring.
What's on the horizon for 2025
In 2024, our email security team focused on refining the user interface and improving the incident investigation experience. Looking ahead to 2025, we plan to introduce additional capabilities that deepen the integration of our email security solution with our SASE platform, delivering enhanced insight and protection against user-based threats.
Configurable browser isolation for email
Our Email Link Isolation feature currently applies to links we consider suspicious. However, we intend to allow customers to add customized configurations to meet their internal policies. This enhancement will provide more granular control over which websites users can access from an email message without using an isolated browser.
Outbound DLP for email
We will be releasing an add-in for Microsoft Outlook that will allow customers to use our DLP engine for inspecting outbound email messages. This client-side application enables customers to configure downstream policies that trigger action when a DLP policy is violated, all while minimizing disruption to existing email infrastructure.
Expanded user risk scoring
Cloudflare will be increasing the signals that feed into our user risk scores. This will enable SOC teams to create more policies within Cloudflare or to take automated actions externally based on the level of risk observed.
These are just a few examples of significant releases that will be coming in 2025. Please stay tuned to the Cloudflare blog where we will be announcing these releases as they happen.
Try Cloudflare Email Security today
We provide all organizations (whether a Cloudflare customer or not) with free access to our Retro Scan tool, allowing them to use our predictive AI models to scan existing inbox messages. Retro Scan will detect and highlight any threats found, enabling organizations to remediate them directly in their email accounts. With these insights, organizations can implement further controls, either using Cloudflare Email Security or their preferred solution, to prevent similar threats from reaching their inboxes in the future.
]]>Introducing Annotated Logger: A Python Package to Aid in Adding Metadata to Logshttps://blog.imc.re/RSSBOX/rss/68d82f32.html2024-12-18T13:31:44.000Z2024-12-18T13:31:44.000Z
What it is
Annotated Logger is a Python package that allows you to decorate functions and classes, which then log when complete and can request a customized logger object, which has additional fields pre-added. GitHub’s Vulnerability Management team created this tool to make it easier to find and filter logs in Splunk.
Why we made it
We have several Python projects that have grown in complexity over the years and have used Splunk to ingest and search those logs. We have always sent our logs in via JSON, which makes it easy to add in extra fields. However, there were a number of fields, like what Git branch was deployed, that we also wanted to send, plus, there were fields, like the CVE name of the vulnerability being processed, that we wanted to add for messages in a given function. Both are possible with the base Python logger, but it’s a lot of manual work repeating the same thing over and over, or building and managing a dictionary of extra fields that are included in every log message.
The Annotated Logger started out as a simple decorator in one of our repositories, but was extracted into a package in its own right as we started to use it in all of our projects. As we’ve continued to use it, its features have grown and been updated.
How and why to use it
Now that I’ve gotten a bit of the backstory out of the way, here’s what it does, why you should use it, and how to configure it for your specific needs. At its simplest, you decorate a function with @annotate_logs() and it will “just work.” If you’d like to dive right in and poke around, the example folder contains examples that fully exercise the features.
@annotate_logs() def split_username(annotated_logger, username): annotated_logger.annotate(username=username) annotated_logger.info(“This is a very important message!”, extra={”important”: True}) return list(username)
>>> split_username("crimsonknave"){"created": 1733349907.7293086, "levelname": "DEBUG", "name": "annotated_logger.example.c499f318-e54b-4f54-9030-a83607fa8519", "message": "start", "action": "__main__:split_username", "branch": "unknown-branch", "annotated": true}{"created": 1733349907.7296104, "levelname": "INFO", "name": "annotated_logger.example.c499f318-e54b-4f54-9030-a83607fa8519", "message": "This is a very important message!", "important": true, "action": "__main__:split_username", "branch": "unknown-branch", "username": "crimsonknave", "annotated": true}{"created": 1733349907.729843, "levelname": "INFO", "name": "annotated_logger.example.c499f318-e54b-4f54-9030-a83607fa8519", "message": "success", "action": "__main__:split_username", "branch": "unknown-branch", "username": "crimsonknave", "success": true, "run_time": "0.0", "count": 12, "annotated": true}['c', 'r', 'i', 'm', 's', 'o', 'n', 'k', 'n', 'a', 'v', 'e']>>>>>> split_username(1){"created": 1733349913.719831, "levelname": "DEBUG", "name": "annotated_logger.example.1c354f32-dc76-4a6a-8082-751106213cbd", "message": "start", "action": "__main__:split_username", "branch": "unknown-branch", "annotated": true}{"created": 1733349913.719936, "levelname": "INFO", "name": "annotated_logger.example.1c354f32-dc76-4a6a-8082-751106213cbd", "message": "This is a very important message!", "important": true, "action": "__main__:split_username", "branch": "unknown-branch", "username": 1, "annotated": true}{"created": 1733349913.7200255, "levelname": "ERROR", "name": "annotated_logger.example.1c354f32-dc76-4a6a-8082-751106213cbd", "message": "Uncaught Exception in logged function", "exc_info": "Traceback (most recent call last):\n File \"/home/crimsonknave/code/annotated-logger/annotated_logger/__init__.py\", line 758, in wrap_function\n result = wrapped(*new_args, **new_kwargs) # pyright: ignore[reportCallIssue]\n File \"<stdin>\", line 5, in split_username\nTypeError: 'int' object is not iterable", "action": "__main__:split_username", "branch": "unknown-branch", "username": 1, "success": false, "exception_title": "'int' object is not iterable", "annotated": true}Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<makefun-gen-0>", line 2, in split_username File "/home/crimsonknave/code/annotated-logger/annotated_logger/__init__.py", line 758, in wrap_function result = wrapped(*new_args, **new_kwargs) # pyright: ignore[reportCallIssue] File "<stdin>", line 5, in split_usernameTypeError: 'int' object is not iterable
There are a few things going on in this example. Let’s break it down piece by piece.
The Annotated Logger requires a small amount of setup to use; specifically, you need to instantiate an instance of the AnnotatedLogger class. This class contains all of the configuration for the loggers.
Here we set the name of the logger. (You will need to update the logging config if your name does not start with annotated_logger or there will be nothing configured to log your messages.)
We also set a branch annotation that will be sent with all log messages.
After that, we create an alias for the decorator. You don’t have to do this, but I find it’s easier to read than @al.annotate_logs().
Now, we decorate and define our method, but this time we’re going to ask the decorator to provide us with a logger object, annotated_logger. This annotated_logger variable can be used just like a standard logger object but has some extra features.
This annotated_logger argument is added by the decorator before calling the decorated method.
The signature of the decorated method is adjusted so that it does not have an annotated_logger parameter (see how it’s called with just name).
There are optional parameters to the decorator that allow type hints to correctly parse the modified signature.
We make use of one of those features right away by calling the annotate method, which will add whatever kwargs we pass to the extra field of all log messages that use the logger.
Any field added as an annotation will be included in each subsequent log message that uses that logger.
You can override an annotation by annotating again with the same name
At last, we send a log message! In this message we also pass in a field that’s only for that log message, in the same way you would when using logger.
In the second call, we passed an int to the name field and list threw an exception.
This exception is logged automatically and then re-raised.
This makes it much easier to know if/when a method ended (unless the process was killed).
Let’s break down each of the fields in the log message:
Field
Source
Description
created
logging
Standard Logging field.
levelname
logging
Standard Logging field.
name
annotated_logger
Logger name (set via class instantiation).
message
logging
Standard Logging field for log content.
action
annotated_logger
Method name the logger was created for.
branch
AnnotatedLogger()
Set from the configuration’s branch annotation.
annotated
annotated_logger
Boolean indicating if the message was sent via Annotated Logger.
important
annotated_logger.info
Annotation set for a specific log message.
username
annotated_logger.annotate
Annotation set by user.
success
annotated_logger
Indicates if the method completed successfully (True/False).
run_time
annotated_logger
Duration of the method execution.
count
annotated_logger
Length of the return value (if applicable).
The success, run_time and count fields are added automatically to the message (“success”) that is logged after a decorated method is completed without an exception being raised.
Under the covers
How it’s implemented
The Annotated Logger interacts with Logging via two main classes: AnnotatedAdapter and AnnotatedFilter. AnnotatedAdapter is a subclass of logging.LoggerAdapter and is what all annotated_logger arguments are instances of. AnnotatedFilter is a subclass of logging.Filter and is where the annotations are actually injected into the log messages. As a user outside of config and plugins, the only part of the code you will only interact with are AnnotatedAdapter in methods and the decorator itself. Each instance of the AnnotatedAdapter class has an AnnotatedFilter instance—the AnnotatedAdapter.annotate method passes those annotations on to the filter where they are stored. When a message is logged, that filter will calculate all the annotations it should have and then update the existing LogRecord object with those annotations.
Because each invocation of a method gets its own AnnotatedAdapter object it also has its own AnnotatedFilter object. This ensures that there is no leaking of annotations from one method call to another.
Type hinting
The Annotated Logger is fully type hinted internally and fully supports type hinting of decorated methods. But a little bit of additional detail is required in the decorator invocation. The annotate_logs method takes a number of optional arguments. For type hinting, _typing_self, _typing_requested, _typing_class and provided are relevant. The three arguments that start with _typing have no impact on the behavior of the decorator and are only used in method signature overrides for type hinting. Setting provided to True tells the decorator that the annotated_logger should not be created and will be provided by the caller (thus the signature shouldn’t be altered).
_typing_self defaults to True as that is how most of my code is written. provided, _typing_class and _typing_requested default to False.
class Example: @annotate_logs(_typing_requested=True) def foo(self, annotated_logger): ...
e = Example() e.foo()
Plugins
There are a number of plugins that come packaged with the Annotated Logger. Plugins allow for the user to hook into two places: when an exception is caught by the decorator and when logging a message. You can create your own plugin by creating a class that defines the filter and uncaught_exception methods (or inherits from annotated_logger.plugins.BasePlugin which provides noop methods for both).
The filter method of a plugin is called when a message is being logged. Plugins are called in the order they are set in the config. They are called by the AnnotatedFilter object of the AnnotatedAdapter and work like any logging.Filter. They take a record argument which is a logging.LogRecord object. They can manipulate that record in any way they want and those modifications will persist. Additionally, just like any logging filter, they can stop a message from being logged by returning False.
The uncaught_exception method of a plugin is called when the decorator catches an exception in the decorated method. It takes two arguments, exception and logger. The logger argument is the annotated_logger for the decorated method. This allows the plugin to annotate the log message stating that there was an uncaught exception that is about to be logged once the plugins have all processed their uncaught_exception methods.
Here is an example of a simple plugin. The plugin inherits from the BasePlugin, which isn’t strictly needed here since it implements both filter and uncaught_exception, but if it didn’t, inheriting from the BasePlugin means that it would fall back to the default noop methods. The plugin has an init so that it can take and store arguments. The filter and uncaught_exception methods will end up with the same result: flagged=True being set if a word matches. But they do it slightly differently, filter is called while a given log message is being processed and so the annotation it adds is directly to that record. While uncaught_exception is called if an exception is raised and not caught during the execution of the decorated method, so it doesn’t have a specific log record to interact with and set an annotation on the logger. The only difference in outcome would be if another plugin emitted a log message during its uncaught_exception method after FlagWordPlugin, in that case, the additional log message would also have flagged=True on it.
from annotated_logger.plugins import BasePlugin
class FlagWordPlugin(BasePlugin): “””Plugin that flags any log message/exception that contains a word in a list.””” def init(self, *wordlist): “””Save the wordlist.””” self.wordlist = wordlist
def filter(self, record):"""Add annotation if the message contains words in the wordlist."""for word in self.wordlist: if word in record.msg: record.flagged = Truedef uncaught_exception(self, exception, logger):"""Add annotation if exception title contains words in the wordlist."""for word in self.wordlist: if word in str(exception) logger.annotate(flagged=True)
Plugins are stored in a list and the order they are added can matter. The BasePlugin is always the first plugin in the list; any that are set in configuration are added after it.
When a log message is being sent the filter methods of each plugin will be called in the order they appear in the list. Because the filter methods often modify the record directly, one filter can break another if, for example, one filter removed or renamed a field that another filter used. Conversely, one filter could expect another to have added or altered a field before its run and would fail if it was ahead of the other filter. Finally, just like in the logging module, the filter method can stop a log from being emitted by returning False. As soon as a filter does so the processing ends and any Plugins later in the list will not have their filter methods called.
If the decorated method raises an exception that is not caught, then the plugins will again execute in order. The most common interaction is plugins attempting to set/modify the same annotation. The BasePlugin and RequestsPlugin both set the exception_title annotation. Since the BasePlugin is always first, the title it sets will be overridden. Other interactions would be one plugin setting an annotation before or after another plugin that emits a log message or sends data to a third-party. In both of those cases the order will impact if the annotation is present or not.
Plugins that come with the Annotated Logger:
GitHubActionsPlugin—Set a level of log messages to also be emitted in actions notation (notice::).
NameAdjusterPlugin—Add a pre/postfix to a name to avoid collisions in any log processing software (source is a field in Splunk, but we often include it as a field and it’s just hidden).
RemoverPlugin—Remove a field. Exclude password/key fields and set an object’s attributes to the log if you want or ignore fields like taskName that are set when running async, but not sync.
NestedRemoverPlugin—Remove a field no matter how deep in a dictionary it is.
RenamerPlugin—Rename one field to another (don’t like levelname and want level, this is how you do that).
RequestsPlugin—Adds a title and status code to the annotations if the exception inherits from requests.exceptions.HTTPError.
When adding the Annotated Logger to an existing project, or one that uses other packages that log messages (flask, django, and so on), you can configure all of the Annotated Logger via dictConfig by supplying a dictConfig compliant dictionary as the config argument when initializing the Annotated Logger class. If, instead, you wish to do this yourself you can pass config=False and reference annotated_logger.DEFAULT_LOGGING_CONFIG to obtain the config that is used when none is provided and alter/extract as needed.
There is one special case where the Annotated Logger will modify the config passed to it: if there is a filter named annotated_filter that entry will be replaced with a reference to a filter that is created by the instance of the Annotated Logger that’s being created. This allows any annotations or other options set to be applied to messages that use that filter. You can instead create a filter that uses the AnnotatedFilter class, but it won’t have any of the config the rest of your logs have.
Notes
dictConfig partly works when merging dictionaries. I have found that some parts of the config are not overwritten, but other parts seem to lose their references. So, I would encourage you to build up a logging config for everything and call it once only. If you pass config, the Annotated Logger will call logging.config.dictConfig on your config after it has the option to add/adjust the config.
The logging_config.py example has a much more detailed breakdown and set of examples.
Pytest mock
Included with the package is a pytest mock to assist in testing for logged messages. I know that there are some strong opinions about testing log messages, and I don’t suggest doing it extensively, or frequently, but sometimes it’s the easiest way to check a loop, or the log message is tied to an alert, and it is important how it’s formatted. In these cases, you can ask for the annotated_logger_mock fixture which will intercept, record and forward all log messages.
def test_logs(annotated_logger_mock): with pytest.raises(KeyError): complicated_method() annotated_logger_mock.assert_logged( "ERROR", # Log level "That's not the right key", # Log message present={"success": False, "key": "bad-key"}, # annotations and their values that are required absent=["fake-annotations"], # annotations that are forbidden count=1 # Number of times log messages should match )
The assert_logged method makes use of pychoir for flexible matching. None of the parameters are required, so feel free to use whichever makes sense. Below is a breakdown of the default and valid values for each parameter.
Parameter
Default Value
Valid Values
Description
level
Matches anything
String or string-based matcher
Log level to check (e.g., “ERROR”).
message
Matches anything
String or string-based matcher
Log message to check.
present
Empty dictionary
Dictionary with string keys and any value
Annotations required in the log.
absent
Empty set
`ALL`, set, or list of strings
Annotations that must not be present in the log.
count
All positive integers
Integer or integer-based matcher
Number of times the log message should match.
The present key is often what makes the mock truly useful. It allows you to require the things you care about and ignore the things you don’t care about. For example, nobody wants their tests to fail because the run_time of a method went from 0.0 to 0.1 or fail because the hostname is different on different test machines. But both of those are useful things to have in the logs. This mock should replace everything you use the caplog fixture for and more.
Other features
Class decorators and persist
Classes can be decorated with @annotate_logs as well. These classes will have an annotated_logger attribute added after the init (I was unable to get it to work inside the __init__). Any decorated methods of that class will have an annotated_logger that’s based on the class logger. Calls to annotate that pass persist=True will set the annotations on the class Annotated Logger and so subsequent calls of any decorated method of that instance will have those annotations. The class instance’s annotated_logger will also have an annotation of class specifying which class the logs are coming from.
Iterators
The Annotated Logger also supports logging iterations of an enumerable object. annotated_logger.iterator will log the start, each step of the iteration, and when the iteration is complete. This can be useful for pagination in an API if your results object is enumerable, logging each time a page is fetched instead of sitting for a long time with no indication if the pages are hanging or there are simply many pages.
By default the iterator method will log the value of each iteration, but this can be disabled by setting value=False. You can also specify the level to log the iterations at if you don’t want the default of info.
Provided
Because each decorated method gets its own annotated_logger calls to other methods will not have any annotations from the caller. Instead of simply passing the annotated_logger object to the method being called, you can specify provided=True in the decorator invocation. This does two things: first, it means that this method won’t have an annotated_logger created and passed automatically, instead it requires that the first argument be an existing annotated_logger, which it will use as a basis for the annotated_logger object it creates for the function. Second, it adds the annotation of subaction and sets the decorated function’s name as its value, the action annotation is preserved as from the method that called and provided the annotated_logger. Annotations are not persisted from a method decorated with provided=True to the method that called it, unless the class of the calling method was decorated and the called action annotated with persist=True, in which case the annotation is set on the annotated_logger of the instance and shared with all methods as is normal for decorated classes.
The most common use of this is with private methods, especially ones created during a refactor to extract some self contained logic. But other uses are for common methods that are called from a number of different places.
Split messages
Long messages wreak havoc on log parsing tools. I’ve encountered cases where the HTML of a 500 error page was too long for Splunk to parse, causing the entire log entry to be discarded and its annotations to go unprocessed. Setting max_length when configuring the Annotated Logger will break long messages into multiple log messages each annotated with split=True, split_complete=False, message_parts=# and message_part=#. The last part of the long message will have split_complete=True when it is logged.
Only messages can be split like this; annotations will not trigger the splitting. However, a plugin could truncate any values with a length over a certain size.
Pre/Post hooks
You can register hooks that are executed before and after the decorated method is called. The pre_call and post_call parameters of the decorator take a reference to a function and will call that function right before passing in the same arguments that the function will be/was called with. This allows the hooks to add annotations and/or log anything that is desired (assuming the decorated function requested an annotated_logger).
Examples of this would be having a set of annotations that annotate fields on a model and a pre_call that sets them in a standard way. Or a post_call that logs if the function left a model in an unsaved state.
Runtime annotations
Most annotations are static, but sometimes you need something that’s dynamic. These are achieved via the RuntimeAnnotationsPlugin in the Annotated Logger config. The RuntimeAnnotationsPlugin takes a dict of names and references to functions. These functions will be called and passed the log record when the plugin’s filter method is invoked just before the log message is emitted. Whatever is returned by the function will be set as the value of the annotation of the log message currently being logged.
A common use case is to annotate a request/correlation id, which identifies all of the log messages that were part of a given API request. For Django, one way to do this is via django-guid.
Tips, tricks and gotchas
When using the decorator in more than one file, it’s useful to do all of the configuration in a file like log.py. That allows you to from project.log import annotate_logs everywhere you want to use it and you know it’s all configured and everything will be using the same setup.
Namespacing your loggers helps when there are two projects that both use the Annotated Logger (a package and a service that uses the package). If you are setting anything via dictConfig you will want to have a single config that has everything for all Annotated Loggers.
In addition to setting a correlation id for the API request being processed, passing the correlation id of the caller and then annotating that will allow you to trace from the logs of service A to the specific logs in Service B that relate to a call made by service A.
Plugins are very flexible. For example:
Send every exception log message to a service like Sentry.
Suppress logs from another package, like Django, that you don’t want to see (assuming you’ve configured Django’s logs to use a filter for your Annotated Logger).
Add annotations for extra information about specific types of exceptions (see the RequestsPlugin).
Set run time annotations on a subset of messages (instead of all messages with RuntimeAnnotationsPlugin)
Questions, feedback and requests
We’d love to hear any questions, comments or requests you might have in an issue. Pull requests welcome as well!
]]>Internationalization and Localization: Bringing Cloudflare Radar to a Global Audiencehttps://blog.imc.re/RSSBOX/rss/94362ba1.html2024-12-16T14:00:00.000Z2024-12-16T14:00:00.000Z
Cloudflare Radar celebrated its fourth birthday in September 2024. As we’ve expanded Radar’s scope over the last four years, the value that it provides as a resource for the global Internet has grown over time, and with Radar data and graphs often appearing in publications and social media around the world, we knew that we needed to make it available in languages beyond English.
Localization is important because most Internet users do not speak English as a first language. According to W3Techs, English usage on the Internet has dropped 8.3 points (57.7% to 49.4%) since January 2023, whereas usage of other languages like Spanish, German, Japanese, Italian, Portuguese and Dutch is steadily increasing. Furthermore, a CSA Research study determined that 65% of Internet users prefer content in their language.
To successfully (and painlessly) localize any product, it must be internationalized first. Internationalization is the process of making a product ready to be translated and adapted into multiple languages and cultures, and it sets the foundation to enable your product to be localized later on at a much faster pace (and at a lower cost, both in time and budget). Below, we review how Cloudflare’s Radar and Globalization teams worked together to deliver a Radar experience spanning twelve languages.
What is localization?
Localization (l10n) is the process of adapting content for a region, including translation, associated imagery, and cultural elements that influence how your content will be perceived. The goal, ideally, is to make the content sound like it was originally written with the region in mind, incorporating relevant cultural nuances instead of merely replacing English with translated text.
Localization includes, among others:
Language: Translation, obviously, but it’s just the beginning.
Tone and message: Localization considers what will resonate with your target audience, not just what’s accurate.
Images: What may be appropriate in one country can be problematic in another (maps, for instance, that tend to include disputed territories).
Date, time, measurement, and number formats: Formats change based on location and may differ even within the same language. In the U.S., the date follows this format: “December 15, 2018.” But in the U.K., that same date would be written like this: “15 December 2018.” Not to mention a constant source of confusion: the month/day/year vs.day/month/year difference:
Pixar movies are a great example of localization. Pixar takes great care to internationalize their movie production process, so they can replace or insert scenes that will resonate with watchers all over the world, not just the US. Let’s consider Inside Out (2015). During the movie, Riley reminisces about playing ice hockey back in Minnesota. Most of the world is not as familiar with ice hockey as in the US, so Pixar wisely decided that they would use soccer elsewhere, allowing a more direct emotional connection with those audiences.
Images: scene from Inside Out (2015), produced by Pixar Animation Studios and Walt Disney Pictures. Copyright Pixar Animation Studios and Walt Disney Pictures. Images used under fair use.
And you don’t have to go to computer animated movies. Here’s an example from The Shining (1980) where the famous “All work and no play makes Jack a dull boy” typewriter scene was localized into all languages differently. The producers, in a pre-Information Technology example of internationalization, shot and cut the localized scene into the local versions of the movie.
Images: scene from The Shining (1980), directed by Stanley Kubrick. Copyright Warner Bros. Pictures. Images used under fair use.
Internationalization
Localization is hard, and no one in the business will tell you otherwise. Fortunately there’s a playbook: the first step to localization is internationalization (i18n). Internationalization is the process of making a product ready to be translated and adapted into multiple languages and cultures. It's a preparatory step that helps with translation and localization. The more you internationalize your code and the more you take into account language and cultural nuances, the easier the localization will be.
Hard-coding and externalization
The first step to internationalize Radar was to assess how many of the localizable strings were hard-coded. Hard coding is the practice of embedding data directly into the source code of a program. Although a convenient and fast way to write your code, it makes it more difficult to change or localize the code later.
Most of the strings that make up the Radar pages used to be hard-coded, so before we could begin translating, externalization had to be done, which is the process of extracting any text that needs to be localized from the code and moving it into separate files.
Hard-coded strings:
import Card from “~/components/Card”;import Chart from “~/components/Chart”;
export default function TrafficChart() { return ( <Card title=”Traffic” description=”Share of HTTP requests” > <Chart /> </Card> ); }
Externalized key placeholders:
import { useTranslation } from "react-i18next";import Card from “~/components/Card”;import Chart from “~/components/Chart”;
There are several benefits to externalizing strings:
It allows translators to work on separate, isolated files that contain only localizable strings
It prevents accidental changes to the code
It allows developers to deploy updates, changes, and fixes without having to recompile or redeploy code for each language every time
If you look at the example below, when the code is compiled or deployed, upon reaching line 10 (on the left), it will find a key named traffic.chart.title. It will then proceed to match that key within the JSON file on the right, finding it on line 1090 and resolving it to “Traffic” for English, "Tráfego" for Portuguese and "トラフィック" for Japanese, doing this for every localized JSON file present in the code.
Pseudo translation
Not all strings are easily found and some are buried deep in the code, sometimes in legacy, inherited code or APIs. Fortunately, there are some strategies that help detect hard-coded strings. This is where pseudo translation comes into play.
Pseudo translation is a process that replaces all characters in a string with similar-looking ones; pseudo translated strings are enclosed within [ ] characters, and some extra characters are added to them to simulate text expansion (more on that later). It is an invaluable tool to help us find any hard coded strings, and to stress test the UI for language readiness and length variability, while still keeping the content mostly readable. For example, this string:
Routing Information
looks like this once pseudo translated:
[R~óútíñg Í~ñfó~rmát~íóñ]
Once pseudo translation is done, any English strings left intact are most likely hard coded or come from other sources. In the screenshot below you can see how ASN, Country, Name and Prefix Count did not get pseudo translated and had to be externalized by the Radar developers. The Globalization team collaborated with the Radar team to report and fix hard-coded text issues, as well as the issues that are mentioned in the next few sections.
Text expansion
Text expansion occurs when translated content from one language to another takes up more space than the original. Sometimes this expansion is horizontal, as English to German can expand up to an average of 35%, Spanish 30%, and French 20%). Asian languages might contract from the English but expand vertically. Interestingly, the fewer characters English has, the more the localized languages tend to expand.
UI designers and developers need to keep this in mind when creating their applications. Thus, one important consideration is to test the design mock-ups with larger texts and plan the UI to accommodate for text expansion. If some English content barely fits within its container, it will most likely not fit in other languages and possibly break the layout.
Here’s an example of the same button in different languages in Radar’s fixed-width sidebar. Since it’s the main navigation, truncating the text is not appropriate and the only viable option is wrapping, which means localized buttons can end up having different heights. Sometimes it’s necessary to trade visual consistency for usability.
String concatenation
In English, you can easily chain-connect words because most words lack inflections. Almost all programming languages are designed using the English language in mind. An old linguist joke goes like: an English teacher: a teacher of English or a teacher from England? Case in point, it would be nightmarish to translate this example:
A lovely little old rectangular green French silver whittling knife
Most Western languages need to connect words with some glue: prepositions, articles, or inflections. This is why, in general, string concatenation (putting together sentences or sentence parts by combining two or more strings) is a terrible practice for localization, even though it seems efficient from a development point of view. You can’t assume that all languages follow the same sentence structure as English. Most languages don’t.
Sentences may need to be completely reversed for them to sound grammatically correct in other languages. This becomes a particularly severe problem when a string doesn’t include a placeholder because it’s assumed to be concatenated at the beginning or the end of the string, such as this:
"is currently categorized as:"
Developers need to make sure to include any placeholders within the string itself, so that translators can easily move them as needed, for instance:
"Distribution of {{botClass}} traffic by IP version"
would look like this in Simplified Chinese (notice how the {{botClass}} placeholder got moved)
"{{botClass}} 流量分布（按 IP 版本）"
String reuse
As with string concatenation, string reuse (using the same string in more than one place and just swapping out the contents of a placeholder) seems efficient if you’re a developer. A problem arises when translating this into gendered languages, such as most European languages. In Spanish, depending on its position and context, a word as simple as “open” standing by itself, could have all these different translations:
Other examples are Custom, Detected, or Disabled, which can have different translations depending on their position within a sentence, their location in the UI, depending on whether they accompany a singular, plural, masculine or feminine noun, so extra entries for these may need to be created in the language files.
Translators will also need to know what will replace the placeholders in strings like the one below, because the surrounding wording may refer to a term that is masculine, feminine, or neutral (for languages that have those, such as German). If a placeholder could be more than one of these (a masculine noun but also a feminine noun), the translation will become grammatically incorrect in at least some of the cases. In the following example, translators would need to know what {link1} and {link2} will be replaced with, so they know which grammatically correct wording to use around them.
Your use of the URL Scanner is subject to our {{link1}}. Any personal data in a submitted URL will be handled in accordance with our {{link2}}.
A better way to do this is to have component placeholders and include the text to be translated for context:
Your use of the URL Scanner is subject to our <link1>Online Service Terms of Use</link1>. Any personal data in a submitted URL will be handled in accordance with our <link2>Privacy Policy</link2>.
Regional considerations
Date formats
Date formats vary greatly from country to country. Not only can’t you assume that all countries use a month/day/year format, but even the day that the week starts may be different based on the country or culture.
Here’s a comparison of Radar’s date picker in American English against European Spanish (which has weeks starting on Mondays instead of Sundays), and against Simplified Chinese (which uses a completely different format for dates).
Intl.DateTimeFormat receives a locale and formatting options that differ from string tokens commonly found on date libraries such as Day.js or Moment.js. Unless you specifically use the localized string tokens on those libraries, the order of the tokens is fixed, along with any characters or delimiters you might add to the format, which poses a problem because the date format parts order should change according to the locale.
Intl.DateTimeFormat handles all that and saves you the trouble of having to add a date formatting dependency to your project and loading library-specific locale resources.
Here’s an example of a generic React component using Intl.DateTimeFormat and react-i18next. The code below will render the date as “Tue, Oct 1, 2024” for American English (en-US) and as “2024年10月1日(火)” for Japanese (ja-JP).
import { useTranslation } from "react-i18next";export default function SomeComponent() { const { i18n } = useTranslation(); const date = new Date("2024-10-01"); return ( <time dateTime={date.toISOString()}> {new Intl.DateTimeFormat(i18n.language, { weekday: "short", month: "short", year: "numeric", day: "numeric", }).format(date)} </time> );}
Number notations
Similarly, different locales use different notations for numbers. In the US and the UK, a period is used as the decimal separator, and a comma as the thousands separator. Instead, other countries use a comma as the decimal separator and a period (or a space) as the thousands separator. Again, it’s not necessary for developers to know all the odds and ends for this, as they can use Intl.NumberFormat.
Here’s an example of a generic React component using Intl.NumberFormat and react-i18next. The code below will render the number as “12,345,678.90” for American English (en-US) and as “12 345 678,90” for Portuguese (pt-PT). Intl.NumberFormat options can be passed to format numbers as decimals, percentages, currencies, etc, and specify things like number of decimal places and rounding strategies.
As of mid-December, regionalized number formatting is not fully implemented on Radar. We expect this to be complete by the end of Q1 2025.
List sorting
When you have a list of items that appears sorted, such as a country list in a dropdown, it’s not enough to simply translate the items. For instance, when translated into Portuguese, “South Africa” becomes “África do Sul”, which means it should then go near the top of the list. Besides that, each language has different sorting requirements, and those go way beyond the A-Z alphabet. For instance, several Asian languages don’t use Latin characters at all, and may get sorted by stroke or character radical order instead.
Here’s an example of a generic React country selector component using String.localeCompare and react-i18next. The code below imports a list of countries with name and alpha-2 code and sorts the options according to the translated country name for the active locale. Intl.Collator options can be passed to localeCompare() for specific sorting needs.
import { useTranslation } from "react-i18next";
import Select from “/components/Select”; import COUNTRIES from “/constants/geo”;
Many of the Radar screens and reports include output from Cloudflare or third-party APIs. Unfortunately, the vast majority of these APIs only output English content. When combining that with the translated part of the site, it may give the impression of a poorly localized site.
To solve this, we take API outputs and map everything into separate files, translate all possible messages, and then display that instead of the original output. But as APIs evolve over time and new messages are added, or existing ones get changed, keeping up with these translations becomes an endless game of “whac-a-mole".
"Address unreachable error when attempting to load page": "Error de dirección inaccesible al intentar cargar la página","Authentication failed": "Autenticación fallida","Browser did not fully start before timeout": "El navegador no se ha iniciado por completo antes de agotar el tiempo de espera.","Certificate and/or SSL error when attempting to load page": "Error de certificado y/o SSL al intentar cargar la página","Crawl took too long to finish": "El rastreo ha tardado demasiado en completarse.","DNS resolution failed": "Error de resolución de DNS","Network connection aborted.": "Conexión de red cancelada"
It should be a best practice for APIs to accept a locale parameter or header, and for engineers to have multiple languages in mind when building these APIs, even if it’s just the error messages. That could save time and resources for any number of clients they might have.
Project setup
Radar is a Remix project running on Cloudflare Pages. While researching ways to implement internationalization, we came across this Remix blog post, and after some experimenting, we decided to go with Sergio Xalambrí’s remix-i18next. We mostly followed the installation instructions found on the repo, with some changes.
We have multiple translation files on every locale folder, one for each data source, to help us maintain strings that come from APIs. Each file can be used to create a namespace for translations, to avoid key collisions, and also to be loaded separately as needed for each route or component.
On remix-i18next’s instructions, you’ll find the concept of backend plugins to achieve the loading of these files, except that you cannot use i18next-fs-backend with Cloudflare Pages because there’s no access to the filesystem. To solve that we used the resources approach, similar to what can be found on this sample remix-i18next with vite setup, but we didn’t want to have to maintain the resources dictionary each time we add new namespaces, so vite’s Glob imports came in handy:
import { serverOnly$ } from "vite-env-only/macros";
This creates a server-side only resources dictionary by importing all JSON files in the locales folder to be passed as the resources property for the i18next configuration in Remix’s entry.server.tsx.
To load namespaces on the client side, we created a Remix resource route that uses the resources dictionary and responds with the namespace object of the requested locale:
import { LoaderFunctionArgs } from "@remix-run/server-runtime";
import i18next from "i18next";import i18nextHttpBackend from "i18next-http-backend";import { initReactI18next } from "react-i18next";import { getInitialNamespaces } from "remix-i18next/client";import { options } from “~/i18n”;
Namespaces on the server side get picked up by the getRouteNamespaces function on entry.server.tsx:
const ns = i18n.getRouteNamespaces(remixContext);
On the client-side, examples suggested that you’d have to declare the namespaces on each useTranslation() hook instance, but we worked around that in Remix’s root.tsx file:
import { useLocation, useMatches } from "@remix-run/react";import { useTranslation } from "react-i18next";import { defaultNS } from “~/i18n”;
This causes the client-side plugin to make calls to the resource route and load the required namespaces.
We also wanted to have the locale in the URL pathname, but not for the default language, so Remix’s optional segments allowed us to do just that. remix-i18next does not have URL locale detection by default, but you can provide your own findLocale function that will receive the request as an argument, and you can then parse the request URL to extract the locale.
Search engine optimization
Once you set up your project for internationalization, you can inform search engines of localized versions of your pages. This allows search engines to display localized results of your website in the same language that is being searched.
You should also localize page titles, descriptions, and relevant Open Graph metadata. To achieve this with Remix and remix-i18next, you use the getFixedT method in route loaders to resolve the translations and return data for the meta export:
import type { LoaderFunctionArgs, MetaArgs } from "@remix-run/server-runtime";import i18n from "~/i18next.server";
export const meta = ({ data }: MetaArgs<typeof loader>) => data.meta;
If you are defining default meta tags in parent routes you may also need to merge the meta objects.
Conclusion
There is hardly ever an absolute when dealing with languages. Your Spanish, your French, your Japanese will be different from someone else’s, even if you grew up next door to each other. Family, education, environment, relationships will season and give color to your language. It is like a family recipe — it’s unique, it feels like home and it’s not negotiable. It does not make it better or worse, it just makes it yours. And yours will always be different from other languages.
Localization is hard. We have seen that there are many things that can and will go sideways, and there are many unknowns that bubble up to the surface in the process. It can also make a product better, as it stress tests the product’s code and design. A tight relationship between the globalization and Radar teams helped make our efforts go more smoothly. In addition, our translators stepped up to the challenge, familiarizing themselves with Radar, analyzing the English content, finding the right translation that will not only resonate with the audience but also fit in the space allotted, constantly checking for context, previous translations, consistency, industry standards, adapting to style guides, tone, and messaging, and after all of that, ultimately acknowledging the fact that there will be people who will disagree (to varying levels of zeal) with their choice of words.
If you haven’t done so already, we encourage you to explore the localized versions of Cloudflare Radar. Click the language drop-down in the upper right corner of the Radar interface and select your language of choice — Radar will be presented in that language until a new selection is made. Have comments or suggestions about the translations? Let us know at radar_localization@cloudflare.com.
]]>我为什么开发了一个专栏功能？https://blog.imc.re/RSSBOX/rss/9e02f6c3.html2024-02-02T16:00:00.000Z2024-02-02T16:00:00.000Z在回访用户的过程中，我发现有不少用户用 wiki 当作专栏来使用，可见专栏确实是一个普遍的需求点，而现有的 wiki 系统并不是专门为此场景设计的，虽然能用，但是不够好用。
[root@master ~]# whereis ls ls: /usr/bin/ls /usr/share/man/man1/ls.1.gz [root@master ~]# ls aaa anaconda-ks.cfg localperl nohup.out perl5 ShiSai2022-1.0-SNAPSHOT.jar spark-yarn-logs zookeeper.out [root@master ~]# cd /usr/bin/ [root@master bin]# ll ls -rwxr-xr-x. 1 root root 117680 10月 31 2018 ls
whatis用来获取命令的简介
1 2
[root@master bin]# whatis ls ls (1) - list directory contents
find用来寻找文件的位置
从根目录下开始，寻找cd这个文件的位置。一共找到了3处。
1 2 3 4 5 6 7
[root@master ~]# find / -name cd /root/.npm/_cacache/index-v5/cd /root/.npm/_cacache/content-v2/sha512/f7/cd /usr/bin/cd [root@master ~]# cd /usr/bin/ [root@master bin]# ls cd cd
[root@master ~]# echo $PS1 [\u@\h \W]\$ [root@master ~]# echo $PS2 > [root@master ~]# mysql -u root -p Enter password: # （输入Passwd123！） Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.7.18 MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.
Type 'help;' or '\h'forhelp. Type '\c' to clear the current input statement.
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper]
测试生产者(producer), 并截图保存结果:
1
kafka-console-producer.sh --broker-list 192.168.152.240:9092 --topic test
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper]. 111 222 333 444 555 666 777 888
cd 目录 # 切换目录的命令使用 cd 命令可以切换目录，比如 cd $MP 就是打开$MP 目录。其中在目录前添加 ./module或不加任何标记 module 表示当前目录，添加 ../module 表示上级目录 ，/module 表示从根目录 / 开始的绝对路径目录。
vi 文件目录 # 在命令行中使用编辑器的命令使用 vi 命令可以在shell中编辑文件，比如 vi /etc/hosts 就是编辑/etc/hosts文件。文件目录的标注方法和cd同样，不存在的文件会直接新建，请注意！使用方向键移动光标，点击 i 键，最下方显示 INSERT 后进入编辑模式点击 Esc 键，然后输入 :wq可以保存文件并退出，其中q表示退出，w表示保存
tar # 解压文件的命令使用tar命令可以压缩/解压文件，hadoop配置中只用到了解压文件，这里给出解压文件的示例命令，其他参数请自行tar --help 或者bing查看。
1 2 3
tar -xvzf 压缩目录文件 -C 解压到的目录 # 例子： tar -xvzf /opt/jdk-8u162-linux-x64.tar.gz -C $MP
# hadoop-3.3.0 need to add PATH for start&stop shell hadoop-3.3.0需要添加PATH变量到启动/关闭脚本中
sed -i "2i HDFS_DATANODE_USER=root"$MP/hadoop/sbin/start-dfs.sh sed -i "2i HADOOP_SECURE_DN_USER=hdfs"$MP/hadoop/sbin/start-dfs.sh sed -i "2i HDFS_NAMENODE_USER=root"$MP/hadoop/sbin/start-dfs.sh sed -i "2i HDFS_SECONDARYNAMENODE_USER=root"$MP/hadoop/sbin/start-dfs.sh
sed -i "2i HDFS_DATANODE_USER=root"$MP/hadoop/sbin/stop-dfs.sh sed -i "2i HADOOP_SECURE_DN_USER=hdfs"$MP/hadoop/sbin/stop-dfs.sh sed -i "2i HDFS_NAMENODE_USER=root"$MP/hadoop/sbin/stop-dfs.sh sed -i "2i HDFS_SECONDARYNAMENODE_USER=root"$MP/hadoop/sbin/stop-dfs.sh
sed -i "2i YARN_RESOURCEMANAGER_USER=root"$MP/hadoop/sbin/start-yarn.sh sed -i "2i HADOOP_SECURE_DN_USER=yarn"$MP/hadoop/sbin/start-yarn.sh sed -i "2i YARN_NODEMANAGER_USER=root"$MP/hadoop/sbin/start-yarn.sh
sed -i "2i YARN_RESOURCEMANAGER_USER=root"$MP/hadoop/sbin/stop-yarn.sh sed -i "2i HADOOP_SECURE_DN_USER=yarn"$MP/hadoop/sbin/stop-yarn.sh sed -i "2i YARN_NODEMANAGER_USER=root"$MP/hadoop/sbin/stop-yarn.sh
虽然已经有相关的项目了，比如 Open Note Block Studio 就已经实现了nbs音乐相关的编辑，以及midi文件的转换。不过这个工具目前只支持MCJE，并不支持MCBE，扩展音域似乎也是通过扩展音符包来实现的，相对繁琐。本文主要记录一下MCBE的playsound调音对应88键平均律的对应数值以及方法，虽然不一定在MCJE适用，但是基本方法应该是差不多的，可以作为参考留档。
Take a piano. The keys begin, the keys end. You know there are eightyeight of them, nobody can tell you any different. They are not infinite. You are infinite. And on these keys the music that you can make is infinite. I like that. That I can live by.
nostr.build media uploader 为Nostr网络提供图像存储上传的网站，可以使用直链查看，不过可惜国内屏蔽了，不然可以当图床用x 不过关键的数据存储是否使用Nostr网络这点打个问号，目前来看大概率不是，还是自托管的，毕竟用的php实现的，还能查看具体的存储用量）估计和一般的图床差不多，只是挂了个nostr的名号。
使用的管理员账户，就是使用用户名或微软账号登录到 Windows 的账号，管理权限无限近似于 Administrator，但在某些文件 / 文件夹的操作上，还是不如真正的 Administrator 好用。
在 Windows 内置的账户中，还有两个比 Administrator 权限更高的账户，一个是肉眼可见的 System 权限，最终的 Boss 则是 TrustedInstaller，其实所有账户中它也不是最高的那个，但剩下的无论通过什么方式，都挖掘不出来了，也许超级账户就是 Microsoft 自己了。
uint256 constant private MAX_UINT256 = 2**256 - 1; mapping (address => uint256) public balances; mapping (address => mapping (address => uint256)) public allowed; /* NOTE: The following variables are OPTIONAL vanities. One does not have to include them. They allow one to customise the token contract & in no way influences the core functionality. Some wallets/interfaces might not even bother to look at this information. */ string public name; //fancy name: eg Simon Bucks uint8 public decimals; //How many decimals to show. string public symbol; //An identifier: eg SBX
functionEIP20( uint256 _initialAmount, string _tokenName, uint8 _decimalUnits, string _tokenSymbol ) public { balances[msg.sender] = _initialAmount; // Give the creator all initial tokens totalSupply = _initialAmount; // Update total supply name = _tokenName; // Set the name for display purposes decimals = _decimalUnits; // Amount of decimals for display purposes symbol = _tokenSymbol; // Set the symbol for display purposes }
mapping(address => address[]) public created; mapping(address => bool) public isEIP20; //verify without having to do a bytecode check. bytes public EIP20ByteCode; // solhint-disable-line var-name-mixedcase
functionEIP20Factory() public { //upon creation of the factory, deploy a EIP20 (parameters are meaningless) and store the bytecode provably. address verifiedToken = createEIP20(10000, "Verify Token", 3, "VTX"); EIP20ByteCode = codeAt(verifiedToken); }
//verifies if a contract that has been deployed is a Human Standard Token. //NOTE: This is a very expensive function, and should only be used in an eth_call. ~800k gas functionverifyEIP20(address _tokenContract) public view returns (bool) { bytes memory fetchedTokenByteCode = codeAt(_tokenContract);
if (fetchedTokenByteCode.length != EIP20ByteCode.length) { returnfalse; //clear mismatch }
//starting iterating through it if lengths match for (uint i = 0; i < fetchedTokenByteCode.length; i++) { if (fetchedTokenByteCode[i] != EIP20ByteCode[i]) { returnfalse; } } returntrue; }
EIP20 newToken = (newEIP20(_initialAmount, _name, _decimals, _symbol)); created[msg.sender].push(address(newToken)); isEIP20[address(newToken)] = true; //the factory will own the created tokens. You must transfer them. newToken.transfer(msg.sender, _initialAmount); returnaddress(newToken); }
//for now, keeping this internal. Ideally there should also be a live version of this that // any contract can use, lib-style. //retrieves the bytecode at a specific address. functioncodeAt(address _addr) internal view returns (bytes outputCode) { assembly { // solhint-disable-line no-inline-assembly // retrieve the size of the code, this needs assembly let size := extcodesize(_addr) // allocate output byte array - this could also be done without assembly // by using outputCode = new bytes(size) outputCode := mload(0x40) // new "memory end" including padding mstore(0x40, add(outputCode, and(add(add(size, 0x20), 0x1f), not(0x1f)))) // store length in memory mstore(outputCode, size) // actually retrieve the code, this needs assembly extcodecopy(_addr, add(outputCode, 0x20), 0, size) } } }
EIP20Interface.sol
// Abstract contract for the full ERC 20 Token standard // https://github.com/ethereum/EIPs/blob/master/EIPS/eip-20.md pragma solidity ^0.4.21;
contract EIP20Interface { /* This is a slight change to the ERC20 base standard. function totalSupply() constant returns (uint256 supply); is replaced with: uint256 public totalSupply; This automatically creates a getter function for the totalSupply. This is moved to the base contract since public getter functions are not currently recognised as an implementation of the matching abstract function by the compiler. */ /// total amount of tokens uint256 public totalSupply;
/// @param _owner The address from which the balance will be retrieved /// @return The balance functionbalanceOf(address _owner) public view returns (uint256 balance);
/// @notice send `_value` token to `_to` from `msg.sender` /// @param _to The address of the recipient /// @param _value The amount of token to be transferred /// @return Whether the transfer was successful or not functiontransfer(address _to, uint256 _value) public returns (bool success);
/// @notice send `_value` token to `_to` from `_from` on the condition it is approved by `_from` /// @param _from The address of the sender /// @param _to The address of the recipient /// @param _value The amount of token to be transferred /// @return Whether the transfer was successful or not functiontransferFrom(address _from, address _to, uint256 _value) public returns (bool success);
/// @notice `msg.sender` approves `_spender` to spend `_value` tokens /// @param _spender The address of the account able to transfer the tokens /// @param _value The amount of tokens to be approved for transfer /// @return Whether the approval was successful or not functionapprove(address _spender, uint256 _value) public returns (bool success);
/// @param _owner The address of the account owning tokens /// @param _spender The address of the account able to transfer the tokens /// @return Amount of remaining tokens allowed to spent functionallowance(address _owner, address _spender) public view returns (uint256 remaining);
Metadata of "eip20" was published successfully. token/EIP20.sol : dweb:/ipfs/QmZemboWkhVyhYMRyCJmjwhw4caenvXS5Mw7Uc5tdE77jP token/EIP20Interface.sol : dweb:/ipfs/QmWQEUAdy5QnnCTGsEtRfedZrVG2xZ77YfKiPE5MVvQiSS metadata.json : dweb:/ipfs/QmWFrJHPJLEWJGAoXpq8MD1pGJsvdcfxhPiW5SDXiwEvm8
This message is used to verify that this feed (feedId:83048545374666752) belongs to me (userId:82913974185265152). Join me in enjoying the next generation information browser https://follow.is.
在 Hadoop 系列框架还没出现之前，数据分析工作已经经历漫长的发展，其中以 BI 系统为主的数据分析，已经有非常成熟和稳定的技术解决方案和生态系统，BI 系统的架构图如下：
BI 又叫商业智能，其包括与传统业务系统的区别在于：业务系统更注重于事务型的数据处理，用来支撑企业的各业务线；而 BI 是将企业中所有数据汇聚成数据仓库（DW）并对其进行分析型操作，其中 Cube 是 BI 的核心模块。Cube 是一额更高层的业务抽象模型，在 Cube 上可以进行上钻、下钻、切片等操作、
BI 系统都是基于关系型数据库，关系型数据库使用 SQL 语句进行操作，但是 SQL 在多维操作相对较弱，所以 Cube 有自己独有的查询语言多维查询语言 —— MDX
大多数的数据库服务厂商都提供 BI 服务，轻易便可搭建出一套 OLAP 分析系统
OLTP 联机事务处理，表现为企业中的应用系统如 OA、CRM、ERP、财务软件等供各部门使用
OLAP 联机分析处理，也叫决策支持系统 DSS，通常进行使用者是企业高管或部门管理者
但是随着互联网发展，BI 系统也暴露除了一些缺点:
BI 系统多以分析业务数据产生结构化数据为主，对于非结构化和半结构化数据处理乏力。例如图片、文本、音频的存储、分析。
Real Time Data：实时数据缓存区，这部分是为了达到实时计算的目的，海量数据接收不可能海量实时入历史数据库，那样会出现建立索引延迟、历史数据碎片文件等问题。因此，有一个实时数据缓存区来存储最近几分钟或者几秒钟的数据。这块可以使用 Kudu 或者 Hbase 等组件来实现。这部分数据会通过 Dumper 来合并到历史数据当中。此处的数据模型和 SDK 端数据模型是保持一致的，都是 Common Data Model，例如 “主 - 谓 - 宾” 模型
Historical Data：历史数据沉浸区，这部分是保存了大量的历史数据，为了实现 Ad-hoc 查询，将自动建立相关索引提高整体历史数据查询效率，从而实现秒级复杂查询百亿条数据的反馈。例如可以使用 HDFS 存储历史数据，此处的数据模型依然 SDK 端数据模型是保持一致的 Common Data Model
Dumper：Dumper 的主要工作就是把最近几秒或者几分钟的实时数据，根据汇聚规则、建立索引，存储到历史存储结构当中，可以使用 map reduce、C、Scala 来撰写，把相关的数据从 Realtime Data 区写入 Historical Data 区
Query Engine：查询引擎，提供统一的对外查询接口和协议（例如 SQL JDBC），把 Realtime Data 和 Historical Data 合并到一起查询，从而实现对于数据实时的 Ad-hoc 查询。例如常见的计算引擎可以使用 presto、impala、clickhouse 等
Realtime model feedback：通过 Edge computing 技术，在边缘端有更多的交互可以做，可以通过在 Realtime Data 去设定规则来对 Edge SDK 端进行控制
Stay connected
 for more information with summary statistics of each proxy.](https://github.blog/wp-content/uploads/2024/12/coding_by_ability.png?w=1024&resize=2300%2C1126)
 for more information with summary statistics of each proxy.](https://github.blog/wp-content/uploads/2024/12/project_management_by_ability.png?w=1024&resize=2316%2C1082)
