The Gradle Blog2024-02-13T03:55:34-05:00https://blog.gradle.org/2023 Lunar Year in Review2024-02-12T00:00:00-05:00https://blog.gradle.org/2023-lunar-new-yearOleg Nenashev
<p>Hi all! On behalf of the Gradle community, we wish you a Happy New Lunar Year! 2023 was a great year for Gradle Build Tool, with many new features and initiatives being released.
Let’s recap the key updates:</p>
<ul>
<li>Gradle 8 baseline with 7 feature packed releases</li>
<li>Kotlin DSL became the new default for Gradle</li>
<li>Configuration Caching, Build Speed and Scalability Improvements</li>
<li>Our First Google Summer of Code and Major League Hacking participation</li>
<li>Gradle Enterprise renaming to Develocity, and Gradle Build Scan enhancements</li>
<li>What’s next, Declarative Gradle and the road towards Gradle 9</li>
</ul>
<p><img src="/images/2024/lunar-new-year/image1.png" alt="2023 Lunar Year in Gradle" title="2023 Lunar Year in Gradle" /></p>
<h2 id="gradle-8">Gradle 8</h2>
<p>In February 2023, the Gradle team announced <a href="https://docs.gradle.org/8.0/release-notes.html">Gradle 8.0</a> - a new release that upgraded Kotlin DSL to v1.8 and greatly improved its compilation speed. Gradle 8.0 expanded integration with Java tools like CodeNarc, PMD, and Foojay Disco API for toolchain management. This major release had more than 40 code contributors from the Gradle community. Thanks, everyone!</p>
<p><img src="/images/2024/lunar-new-year/image2.png" alt="Gradle 8.0 release" title="Gradle 8.0 release" /></p>
<p>See <a href="https://gradle.org/whats-new/gradle-8/">this blog post</a> for the full list of the introduced features and the migration guidelines.
After the 8.0 release, we had 6 more major releases with many more features.
Beyond the key updates listed below,
you can find more in the changelogs and our <a href="https://newsletter.gradle.org/">newsletter</a>.</p>
<h2 id="kotlin-dsl-is-the-new-default">Kotlin DSL is the New Default</h2>
<p>In April 2023, <strong>Kotlin DSL</strong> <a href="https://blog.gradle.org/kotlin-dsl-is-now-the-default-for-new-gradle-builds">became</a> the default recommended option for new Gradle projects. Kotlin DSL for Gradle uses the Kotlin language to enable full IDE assistance for build authoring in IntelliJ IDEA and Android Studio. This includes auto-completion, smart content assist, quick access to documentation, navigation to source, and context-aware refactoring. With Kotlin DSL, you can edit your build logic with the same editing experience you are used to when working with your production and test code. Groovy DSL remains fully supported, and there are no plans for deprecation. <a href="https://docs.gradle.org/current/userguide/kotlin_dsl.html">Learn More</a> about Kotlin DSL in Gradle.</p>
<p><img src="/images/2024/lunar-new-year/image3.png" alt="Kotlin DSL is Now the Default for New Gradle Builds" title="Kotlin DSL is Now the Default for New Gradle Builds" /></p>
<p>Culminating efforts around Kotlin DSL in Gradle and its massive adoption in 2023, Gradle joined the <a href="https://kotlinfoundation.org/">Kotlin Foundation</a> as a silver member. See the full press release <a href="https://gradle.com/press-media/gradle-inc-joins-kotlin-foundation-as-first-new-member-since-founding-by-google-and-jetbrains/">here</a>. We now participate in many technical and outreach initiatives the foundation runs.</p>
<p><img src="/images/2024/lunar-new-year/image4.png" alt="New members of the Kotlin Foundation in 2023" title="New members of the Kotlin Foundation in 2023" /></p>
<h2 id="configuration-caching">Configuration Caching</h2>
<p>In Gradle 8.1 (April 2023), we also announced the general availability of the <a href="https://docs.gradle.org/8.5/userguide/configuration_cache.html">Gradle Configuration Cache</a>. This feature significantly improves build performance by caching the result of the configuration phase and reusing this for subsequent builds. Using the configuration cache, Gradle can skip the configuration phase entirely when nothing that affects the build configuration, such as build scripts, has changed. See this <a href="https://blog.gradle.org/improvements-in-the-build-configuration-input-tracking">blog post</a> by Mikhail Lopatkin to learn more about how configuration caching works.</p>
<p><img src="/images/2024/lunar-new-year/image5.gif" alt="Gradle Configuration Cache in Action" title="Gradle Configuration Cache in Action" /></p>
<h2 id="gradle-build-speed-and-scalability">Gradle Build Speed and Scalability</h2>
<p>In addition to configuration caching, Gradle Build Tool received many updates to improve developer velocity at scale, especially in the 8.2 and 8.3 releases at the beginning of the year. We added support for <a href="https://docs.gradle.org/current/userguide/gradle_daemon.html">Persistent Compiler Daemons</a> to reduce initialization times. We were able to drastically reduce memory consumption and, at the same time, reduce build times:</p>
<p>Memory consumption reductions on a sample Android Studio project:</p>
<p><img src="/images/2024/lunar-new-year/image6.png" alt="Memory consumption reductions on a sample Android Studio project" title="Memory consumption reductions on a sample Android Studio project" /></p>
<p>Build times in different Gradle versions and Bazel:</p>
<p><img src="/images/2024/lunar-new-year/image7.png" alt="Build times in different Gradle versions and Bazel" title="Build times in different Gradle versions and Bazel" /></p>
<p>To learn more about performance improvements, see this <a href="https://blog.gradle.org/our-approach-to-faster-compilation">blog post</a>.</p>
<h2 id="documentation-updates">Documentation Updates</h2>
<p>As you may have noticed, the <a href="https://docs.gradle.org/current/userguide/userguide.html">Gradle User Manual</a> got a significant update this year, including but not limited to content refresh, new examples, and navigation experience. There are many new pages, including new solution pages for technologies and IDEs. Take a look!</p>
<p><img src="/images/2024/lunar-new-year/image8.png" alt="Gradle User Manual - new documentation page" title="Gradle User Manual - new documentation page" /></p>
<p>In the recent Gradle 8.6 release, we also introduced the Dark Theme for Gradle documentation, and there are plans to continue improving the documentation engine and simplify feedback and contribution to the docs. We have a #docs channel on the Gradle Community Slack, and we welcome anyone to discuss the content and documentation experience there.</p>
<h2 id="our-first-google-summer-of-code">Our First Google Summer of Code</h2>
<p>In the summer of 2023, The Kotlin Foundation sponsored four projects for the Google Summer of Code 2023. The Gradle Kotlin Script Support for Eclipse and BuildShip project by <a href="https://www.linkedin.com/in/vladimir0v/">Nikolai Vladimirov</a> was one of them! Gradle, a Kotlin Foundation member, and <a href="https://github.com/donat">Donát Csikós</a> provided mentoring support for this project.</p>
<p>The project’s initial goal was to improve user interaction with Gradle Kotlin scripts inside the Eclipse IDE. The project aimed to implement several key functionalities, including syntax highlighting for Kotlin scripts, content assistance, go-to-definition, accurate diagnostics for syntax errors, signature help, and hovering. Nikolai successfully delivered these features by implementing support for Kotlin scripts based on the existing <a href="https://github.com/fwcd/kotlin-language-server">kotlin-language-server</a> project, creating a new system to separate compilation environments for each build script. <a href="https://kotlinfoundation.org/news/gsoc-2023-eclipse-gradle-kotlin/">Read more</a> about this project by Nikolai and the team.</p>
<p><img src="/images/2024/lunar-new-year/image9.png" alt="Syntax highlighting for .kts scripts in Eclipse IDE" title="Syntax highlighting for .kts scripts in Eclipse IDE" /></p>
<p>In 2023, we also sponsored two Major League Hacking interns who improved documentation and made various improvements in Gradle Build Tool, including better error handling for the project. See the <a href="https://news.mlh.io/major-league-hacking-partners-with-gradle-to-empower-the-next-generation-of-open-source-leaders-09-28-2023">blog post</a> for details.</p>
<p>And stay tuned for Google Summer of Code 2024!
We have submitted a few project ideas to the Kotlin Foundation and to the Eclipse Foundation.</p>
<h2 id="gradle-enterprise-renaming-welcome-develocity">Gradle Enterprise Renaming, Welcome Develocity</h2>
<p>For Develocity, formerly known as Gradle Enterprise, it was also a packed year. First, Gradle Enterprise became <strong>Develocity</strong>, addressing the naming confusion reported by many community members.
See <a href="https://gradle.com/blog/develocity-year-in-review-2023/">this blog post</a>
for the full list of Develocity and Gradle Build Scan updates.</p>
<p><img src="/images/2024/lunar-new-year/image10.png" alt="Develocity 2023 update" title="Develocity 2023 update" /></p>
<h2 id="gradle-build-scan">Gradle Build Scan</h2>
<p>There are also many enhancements in <a href="https://scans.gradle.com/">Gradle Build Scan</a>
which now provides even more insights for Gradle.
With the new versions you can:</p>
<ul>
<li><a href="https://gradle.com/develocity/releases/2023.4#insights-into-gradles-artifact-transform-executions">Get insights into Gradle’s artifact transform executions</a></li>
<li><a href="https://gradle.com/develocity/releases/2023.3#observe-build-time-cost-of-dependency-downloading-for-gradle-and-apache-maven">Analyze build time cost of dependency downloading</a></li>
</ul>
<p><img src="/images/2024/lunar-new-year/gradle_build_scan.png" alt="Gradle Build Scan" title="Gradle Build Scan" /></p>
<p>The test improvements for this year were great as well!
In particular, we added
<a href="https://gradle.com/develocity/releases/2023.4#detection-of-flaky-tests-across-builds">Detection of flaky tests</a> and
<a href="https://gradle.com/develocity/releases/2023.2#fine-grained-test-distribution-observability-in-build-scans">Fine-grained Test Distribution observability</a>
to the build scans.
Those features depend on the build history and hence require a full-fledged Develocity instance.</p>
<h2 id="enters-declarative-gradle">Enters Declarative Gradle</h2>
<p>This year, we also announced the <strong>Declarative Gradle</strong> project. The goal is to deliver an elegant and extensible declarative build language that allows developers to describe any software clearly and understandably.
The Gradle Team drives this project and collaborates closely with Google and JetBrains, which also runs an experimental project called Amper.
See <a href="https://blog.gradle.org/declarative-gradle">this blog post</a> by Piotr Jagielski to learn more about the project and its goals.</p>
<p>Here is how Declarative Gradle top-level definitions in Kotlin DSL may look like:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Clear type of software the project produces</span>
<span class="nf">javaLibrary</span> <span class="p">{</span>
<span class="c1">// GroupID/ArtifactID/Version for publishing</span>
<span class="nf">publishedAs</span><span class="p">(</span><span class="s">"my-group:my-lib:2.0"</span><span class="p">)</span>
<span class="c1">// Common dependencies for all targets</span>
<span class="nf">dependencies</span> <span class="p">{</span>
<span class="nf">api</span><span class="p">(</span><span class="s">"some:lib:1.2"</span><span class="p">)</span>
<span class="nf">implementation</span><span class="p">(</span><span class="n">projects</span><span class="p">.</span><span class="n">someLib</span><span class="p">)</span>
<span class="p">}</span>
<span class="c1">// A library might have more than one target</span>
<span class="nf">targets</span> <span class="p">{</span>
<span class="c1">// All information about specific targets is grouped here</span>
<span class="nf">java</span><span class="p">(</span><span class="mi">11</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Specific information about Java 11 target</span>
<span class="nf">dependencies</span> <span class="p">{</span>
<span class="nf">implementation</span><span class="p">(</span><span class="s">"some:back-port-lib:1.5"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nf">java</span><span class="p">(</span><span class="mi">21</span><span class="p">)</span> <span class="c1">// No additional info for the Java 21 target</span>
<span class="p">}</span>
<span class="nf">tests</span> <span class="p">{</span>
<span class="nf">unit</span> <span class="p">{</span>
<span class="c1">// Dependencies for the unit tests</span>
<span class="nf">dependencies</span> <span class="p">{</span>
<span class="nf">implementation</span><span class="p">(</span><span class="s">"some:other-lib:1.4"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>You can find some prototypes and experimental work in the <a href="https://github.com/gradle/declarative-prototypes">gradle/declarative-prototypes</a> repository.
Work in progress, and any feedback is welcome!
See the README file for discussion and feedback links.</p>
<h2 id="whats-next">What’s Next?</h2>
<p>In the following months, our key priorities remain improving the developer experience for Gradle users, particularly Declarative Gradle, and improving error reporting that will help simplify mitigation of stricter task dependency management introduced in Gradle 8.x. We also plan to provide <a href="https://gradle.github.io/configuration-cache/#project_isolation">better project isolation</a>, extending the configuration caching and improving performance, particularly for the synchronization of Android Studio and IntelliJ IDEA projects.</p>
<p>On February 06, Louis Jacomet, a lead software engineer and the support team leader at Gradle, spoke at Jfokus about the road to Gradle 9 and our focus on scalability and comprehensibility. Louis highlighted the following:</p>
<ul>
<li>Continued investment in configuration caching</li>
<li>Upcoming isolated projects feature</li>
<li>Higher-level modeling of software projects</li>
<li>Separation of concerns for building users and authors</li>
</ul>
<p>Many of those items are already on the <a href="https://github.com/orgs/gradle/projects/31">Gradle Build Tool public roadmap</a>, and you can find the information and links there. We will keep posting updates on the evolution of Gradle 9 in the following newsletters.</p>
<p>You can find the slides from Louis <a href="https://github.com/ljacomet/jfokus-gradle9">here</a>. The video recording will be published later on <a href="https://www.jfokus.se/talks/1709">the Jfokus page</a> and YouTube channel. We also plan to publish the text version on the Gradle blog soon. Stay tuned!</p>
<p><img src="/images/2024/lunar-new-year/image11.jpeg" alt="The Road to Gradle 9" title="The Road to Gradle 9" /></p>
<h2 id="how-do-i-participate">How Do I Participate?</h2>
<p>If you want to share feedback or join any of these initiatives, please join us on the <a href="https://gradle.org/resources/">community Slack channel</a>! Our GitHub issues and Community forums are also open 24/7.</p>
<p>Check out the Gradle Build Tool <a href="https://github.com/orgs/gradle/projects/31">public roadmap</a> to learn more about what is coming soon in the next releases!</p>
<p><strong>And have a Gradlest New Year!</strong></p>
Declarative Gradle2023-11-10T00:00:00-05:00https://blog.gradle.org/declarative-gradlePiotr Jagielski
<p>Part of our vision for Gradle Build Tool is to deliver an elegant and extensible declarative build language that allows developers to describe any kind of software in a clear and understandable way.</p>
<p>Gradle’s build language is already extensible in the most fundamental ways, which results in a high degree of flexibility. This is one of the main reasons Gradle is the build system of choice for many companies, including large enterprises, and the default build system for Android and Kotlin Multiplatform.</p>
<p>However, our build language is currently not always fully declarative, clear, and understandable. While it has long been a best practice that build scripts should be declarative, and the build logic should be kept in plugins, this is not the reality for many projects. We’ve seen projects in the wild that mix declarative and imperative code and make build scripts that are long and complex. Gradle-specific concepts used in build scripts are not always familiar to software developers. This can make Gradle less approachable for developers unfamiliar with Gradle. At the same time, it makes it difficult for the IDEs to offer reliable support for editing build scripts.</p>
<p>This blog post explains the Gradle team’s perspective for what we call a developer-first software definition, or Declarative Gradle for short. This post outlines our plans for making the “elegant and declarative build language” part of our vision a reality. </p>
<h1 id="software-developers-and-build-engineers">Software Developers and Build Engineers</h1>
<p>Let’s first clarify the distinction between the two largest groups of Gradle users: software developers and build engineers. </p>
<p>The majority of Gradle Build Tool users are software developers. Their job is to improve their software product by shipping features, fixing bugs, etc. Software developers don’t start their day with the intention of running builds. While they rely on the build system to complete their tasks, it’s not their primary focus.</p>
<p>The second group of users are build engineers. Unlike software developers, their responsibility is to maintain the build itself. This group has historically appreciated Gradle’s flexibility to address even the most complex build automation requirements. </p>
<p>In practice, there’s often overlap between the two roles. In smaller teams, a few engineers familiar with Gradle tend to own the build. In larger teams, there are often dedicated build engineers.</p>
<h1 id="software-definition-vs-build-logic">Software Definition vs Build Logic</h1>
<p>Next, let’s clarify the distinction between software definition and build logic. </p>
<p><strong>Software definition</strong> is <em>what</em> needs to be built. It defines the structure of the software being produced and includes information such as:</p>
<ul>
<li>What kind of software is being produced?</li>
<li>Which languages is the software implemented in?</li>
<li>Which platforms does the software target?</li>
<li>What are the dependencies of the software?</li>
<li>Which toolchains should be used to compile/link/instrument/bundle/document the software?</li>
<li>What quality checks need to be done before the software can be released?</li>
</ul>
<p><strong>Build logic</strong> is <em>how</em> the software will be built. It is the code that adds new capabilities to Gradle, integrates different tools, and supplies conventions to the software definition. </p>
<p>The software definition is meant to be read and modified by software developers. Build logic is intended to be read and modified by build engineers.</p>
<p>In Gradle, the software definition lives in Groovy DSL and Kotlin DSL build scripts and settings. It is recommended that build logic is kept within plugins. Nevertheless, users have the flexibility to include build logic within build scripts alongside software definitions. This has resulted in a blurring of the boundaries between the concerns and requirements of software developers and build engineers. </p>
<h1 id="what-is-developer-first-software-definition">What Is Developer-first Software Definition</h1>
<p>Our strategic goal is to make the software definition clearer and simpler for developers. </p>
<p>We see two key steps to achieve this: </p>
<h2 id="separate-the-software-definition-and-build-logic-with-restricted-dsl">Separate the software definition and build logic with restricted DSL</h2>
<p>As mentioned above, mixing software definition and build logic is not recommended. </p>
<p>We plan to provide a restricted DSL that separates the software definition and build logic so that the build language is fully declarative. This will effectively enforce existing best practices. </p>
<p>The restricted DSL will allow only a limited set of constructs, such as nesting blocks, assigning values, and selected method invocations. Generic control flow and calls to arbitrary methods will be disallowed. You will be able to write your build logic in any JVM language, such as Java, Kotlin, or Groovy, but that logic will reside in plugins (either local or published).</p>
<p>This will provide a better separation between the concerns of software developers and build engineers, as well as a simpler view of the build without limiting the tool’s flexibility. </p>
<h2 id="match-the-software-definition-to-the-software-domain">Match the software definition to the software domain</h2>
<p>While the restricted DSL is a step in the right direction, it is only a partial solution to make the software definition truly developer-first.</p>
<p>The other problem is that the software definition currently does not use concepts that reflect the underlying software domain in all cases. To work with Gradle, developers typically need to understand Gradle-specific building blocks, such as dependency configurations and tasks. These concepts are specific to build logic and should therefore only concern build engineers. In most cases, software developers should be able to configure everything they need by using concepts they are already familiar with, such as libraries, applications, versions, and test suites. </p>
<p>The Android Gradle Plugin is a good example of a more declarative software definition DSL based on Gradle. While the broader JVM ecosystem hasn’t reached our desired state, recent features, such as <a href="https://docs.gradle.org/current/userguide/jvm_test_suite_plugin.html">test suites</a> and <a href="https://docs.gradle.org/current/userguide/toolchains.html">toolchains</a>, were a notable step in this direction. </p>
<p>One of our goals is to provide a broader set of such high-level constructs that can be shared across ecosystems. This will make existing ecosystems more consistent, such as Java, Android, and KMP. It will also make it much easier to implement support for new ecosystems, such as iOS or .NET, in the future.</p>
<h1 id="ide-experience">IDE Experience</h1>
<p>The crucial aspect of the developer-first software definition is that it makes providing an excellent IDE experience easier. </p>
<p>The software definition is traditionally edited by hand. With Groovy DSL, only rudimentary IDE assistance was possible because of the dynamic nature of the Groovy language. With Kotlin DSL, excellent support is available in supported IDEs such as IntelliJ IDEA and Android Studio. This includes auto-completion, smart content assist, quick access to documentation, navigation to source, and context-aware refactoring. With Kotlin DSL, you can edit your build logic with the same editing experience you are used to when working with your production code.</p>
<p>However, it is difficult for IDE vendors to automatically change the software definition or create a UI editor when arbitrary code is possible in build scripts. By making the software definition fully declarative, support will be much easier. We know that IDE vendors are very interested in providing such capability. With that, software developers can choose the mode of editing (UI or by hand) based on their preferences and level of expertise. </p>
<p>There are also performance considerations. For example, the IDE sync process involves a series of steps, including complex calculations such as dependency resolution, to provide a complete view of the project in the IDE. With a fully declarative software definition that can be parsed swiftly and reliably, we can promptly provide an initial view of the project in the IDE, even for large projects. </p>
<h1 id="collaboration-between-gradle-google-and-jetbrains">Collaboration Between Gradle, Google and JetBrains</h1>
<p>Gradle, Google, and JetBrains have a long and fruitful history of collaboration. Our last major joint effort resulted in <a href="https://blog.gradle.org/kotlin-dsl-is-now-the-default-for-new-gradle-builds">Kotlin DSL Becoming the Default for New Gradle Builds</a>. This triggered an explosion in the popularity of Kotlin DSL and enabled many users to benefit from the full IDE assistance for Gradle build scripts in the supported IDEs.</p>
<p>When we met in person this October with representatives from all three companies, we put most of our attention into declarative software definition. We all recognize the existing constraints to user experience resulting from the current mixture of software definition with build logic and lower-level constructs in many projects. Gradle, Google, and JetBrains are collaborating on delivering a solution to those problems. We are working closely to ensure the solution will work well for Android and Kotlin Multiplatform ecosystems with excellent IDE support. </p>
<p>Gradle Team drives this project and collaborates closely with JetBrains and Google to ensure excellent IDE support in IntelliJ IDEA and Android Studio. This project is one of the top priorities for Gradle. We are already intensively working on an initial prototype. Early next year, we plan to share an initial solution for simple projects with early adopters, starting with Android. </p>
<p>At the same time, JetBrains is running an experiment with similar goals called <a href="https://blog.jetbrains.com/blog/2023/11/09/amper-improving-the-build-tooling-user-experience/">Amper</a>. One of our objectives is to leverage this experiment to benefit the development of Declarative Gradle and provide a great user experience for Kotlin and Kotlin Multiplatform. Therefore, in addition to the dedicated group focused on Declarative Gradle, we are working closely with the Amper team at JetBrains. </p>
<p>We hope this collaboration will improve user experience across multiple ecosystems, including Java, Android, and Kotlin Multiplatform. </p>
<h1 id="transition">Transition </h1>
<p>We are excited about the developer-first software definition and strongly believe it will make both software developers and build engineers using Gradle happier. At the same time, we are mindful of billions of lines of code that are currently not compatible with this approach. </p>
<p>One of our goals is to make it easy for teams to migrate to the new software definition incrementally once it’s available. We are currently investigating using a restricted DSL syntax that would be compatible with existing declarative Kotlin DSL scripts. The restricted DSL will be an additional option next to Groovy and Kotlin DSL. The existing DSL will continue to be fully supported, and we are exploring ways to allow incremental migration of build files to restricted DSL. We are also looking into tooling that could assist with the transition. </p>
<h1 id="feedback">Feedback</h1>
<p>We’d love to know how you feel about our vision and plans for developer-first software definition. As always, let us know if you have any questions or feedback on our <a href="https://discuss.gradle.org/">forums</a> or <a href="https://gradle.com/slack-invite">Community Slack</a>.</p>
Improvements in the Build Configuration Input Tracking2023-09-01T00:00:00-04:00https://blog.gradle.org/improvements-in-the-build-configuration-input-trackingMikhail Lopatkin
<p>The <a href="https://docs.gradle.org/current/userguide/configuration_cache.html">configuration cache</a> is a feature that significantly improves build performance by caching the result of the configuration phase and reusing it for subsequent builds. Using the configuration cache, the Gradle Build Tool can skip the configuration phase entirely when nothing that affects the build configuration, such as build scripts, has changed.</p>
<p>In Gradle 8.1, the configuration cache became stable and recommended for adoption. Stability, in this case, means that the behavior is finalized, and all breaking changes follow the Gradle <a href="https://docs.gradle.org/current/userguide/feature_lifecycle.html">deprecation process</a>. In general, if something works now, it will continue to work the same way—unless the behavior is buggy and may lead to incorrect builds. While some features are <a href="https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:not_yet_implemented">not yet implemented</a>, most users can already benefit from the speed-ups brought by the configuration cache.</p>
<h2 id="build-configuration-inputs">Build configuration inputs</h2>
<p>When writing configuration-cache-compatible build logic or plugins, it is important to understand what Gradle considers build configuration inputs and how they affect cache invalidation.</p>
<p>The concept is similar to <a href="https://docs.gradle.org/current/userguide/build_cache.html">task caching</a>:</p>
<ol>
<li>Everything that contributes to the build logic execution is an input.</li>
<li>The configuration phase is an action.</li>
<li>The resulting task execution graph is the output.</li>
</ol>
<p><img src="/images/improvements-in-the-build-configuration-input-tracking/task_vs_configuration_inputs.png" alt="Task inputs and outputs compared to the configuration phase inputs and outputs" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>Examples of the build configuration inputs are:</p>
<ul>
<li>Build scripts and their dependencies (plugins and libraries).</li>
<li>Environment variables and system properties read at configuration time.</li>
<li>Files read by the build logic or plugins at configuration time.</li>
<li>Information about the file system (existence of files, directory structure) obtained at configuration time.</li>
<li>The output of external processes executed at configuration time.</li>
</ul>
<p>When build logic or a plugin reaches out to something “environmental” at configuration time, the obtained data becomes part of the configuration cache fingerprint. Next time the build runs, Gradle checks if the values of the inputs are still the same—it re-reads environment variables, system properties, files, and re-runs external processes. Then, it compares them with what was seen during the first run when the cache entry was stored. If everything matches, the cache entry is reused, and the task graph is loaded from the cache and executed without running the configuration phase.</p>
<p>It is important to correctly detect all inputs because otherwise, the build using the cache can be incorrect—its result won’t match the non-cached one. For example, if build logic checks for the existence of a file at configuration time and configures a task differently depending on this, then the configuration cache should be invalidated if the file appears or disappears.</p>
<p>To ensure correctness, Gradle employs several techniques:</p>
<ul>
<li>Provide APIs for build authors to explicitly specify values as coming from the environment.</li>
<li>Forbid uses of incompatible APIs when configuration cache is enabled.</li>
<li>Rewrite configuration logic bytecode to intercept environment access.</li>
</ul>
<p>We continuously improve build input tracking with every release so that Gradle can reliably detect more types of inputs. These newly discovered inputs usually do not negatively affect existing builds utilizing the configuration cache. The build still reuses the cache when the new input remains unchanged. In that sense, detecting new kinds of inputs is not a breaking change but more of a bug fix that eliminates false cache hits. However, certain patterns may lead to a slightly different user experience, requiring a few additional non-cacheable runs before the cached configuration can be used in all subsequent builds. For example, the initial build may create some files after their absence has been recorded, making the cache entry invalid for the next run. Such behavior can be problematic for short-lived environments like ephemeral CI builds.</p>
<h2 id="temporarily-ignoring-build-configuration-inputs">Temporarily ignoring build configuration inputs</h2>
<p>Adapting plugins and build logic to avoid unnecessary cache invalidations takes time. However, the input can be ignored for some scenarios without affecting the cache correctness. As an interim solution, alongside the newly introduced input detection, Gradle provides a way for build users to suppress it with <a href="https://docs.gradle.org/nightly/userguide/configuration_cache.html#config_cache:adoption:changes_in_behavior">Gradle properties</a>, on a per-input basis whenever possible.</p>
<p>Plugin authors can document the recommended suppressions while working on an update<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>, but there is no way to apply suppressions from the plugin code. As with the general Gradle deprecation process, the option to ignore the input will be available at least until the next major release.</p>
<p>Below we discuss two major examples of configuration input checks that improve configuration cache correctness, how they can be temporarily disabled, and how to properly fix build logic or plugin code to work with them.</p>
<h2 id="use-case-1-file-api-tracking">Use case 1: File API tracking</h2>
<p>Gradle 8.1 introduced tracking of <code class="language-plaintext highlighter-rouge">File.exists</code>, <code class="language-plaintext highlighter-rouge">File.isDirectory</code>, <code class="language-plaintext highlighter-rouge">File.list</code>, and similar file system-querying calls. This helps to ensure correct build results in case the shape of the task graph depends on the presence of a file, directory, etc.</p>
<p>However, the new check might be problematic for plugins that use the “check then write” pattern:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(!</span><span class="n">someFile</span><span class="o">.</span><span class="na">exists</span><span class="o">())</span> <span class="o">{</span>
<span class="n">someFile</span><span class="o">.</span><span class="na">createNewFile</span><span class="o">();</span>
<span class="o">}</span>
</code></pre></div></div>
<p>That’s because when the checked file doesn’t exist, Gradle records its absence as a configuration input and stores this in the cache. Upon the next build run, Gradle discovers that the file now exists, and therefore the cache has to be invalidated, even though the result of the build configuration is likely the same. Input detection cannot infer how checking the existence of a file affects build logic. In this case, the build or plugin author has to explicitly declare their intentions by using the appropriate Gradle API.</p>
<h3 id="temporarily-disabling-file-api-tracking">Temporarily disabling file API tracking</h3>
<p>While waiting for the updated plugin, build users can disable these checks for selected files with the <a href="https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:adoption:changes_in_behavior"><code class="language-plaintext highlighter-rouge">org.gradle.configuration-cache.inputs.unsafe.ignore.file-system-checks</code> property</a>. We don’t recommend doing so without assessing the impact first. Note that the check was added in Gradle 8.1, and disabling is only possible in Gradle 8.3 onward.</p>
<h3 id="fixing-plugins-or-build-logic-to-work-with-api-tracking">Fixing plugins or build logic to work with API tracking</h3>
<p>Let’s see how some uses of file API can be rewritten in a configuration-cache compatible way.</p>
<h4 id="reading-and-writing-an-optional-configuration-file">Reading and writing an optional configuration file</h4>
<p>Suppose the plugin in question reads a configuration file. The plugin creates the file with a default configuration if it doesn’t exist. The user can modify it if needed. A value from the file is then used to conditionally register a task:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">loadConfiguration</span><span class="p">(</span><span class="n">configFile</span><span class="p">:</span> <span class="nc">File</span><span class="p">):</span> <span class="nc">Properties</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">result</span> <span class="p">=</span> <span class="nc">Properties</span><span class="p">()</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">configFile</span><span class="p">.</span><span class="nf">exists</span><span class="p">())</span> <span class="p">{</span>
<span class="n">result</span><span class="p">[</span><span class="s">"some.property"</span><span class="p">]</span> <span class="p">=</span> <span class="s">"default value"</span>
<span class="n">configFile</span><span class="p">.</span><span class="nf">bufferedWriter</span><span class="p">().</span><span class="nf">use</span> <span class="p">{</span>
<span class="n">result</span><span class="p">.</span><span class="nf">store</span><span class="p">(</span><span class="n">it</span><span class="p">,</span> <span class="s">""</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">result</span>
<span class="p">}</span>
<span class="n">configFile</span><span class="p">.</span><span class="nf">bufferedReader</span><span class="p">().</span><span class="nf">use</span> <span class="p">{</span> <span class="n">result</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">it</span><span class="p">)</span> <span class="p">}</span>
<span class="k">return</span> <span class="n">result</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nf">loadConfiguration</span><span class="p">().</span><span class="nf">getProperty</span><span class="p">(</span><span class="s">"should.register.task"</span><span class="p">).</span><span class="nf">toBoolean</span><span class="p">())</span> <span class="p">{</span>
<span class="n">tasks</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="s">"someTask"</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In this case, the next build run will not reuse the configuration cache entry because the configuration file, which was absent at check time, is now present. This is a perfect use case for a <a href="https://docs.gradle.org/current/javadoc/org/gradle/api/provider/ValueSource.html">ValueSource</a>:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">abstract</span> <span class="kd">class</span> <span class="nc">PropertiesValueSource</span>
<span class="p">:</span> <span class="nc">ValueSource</span><span class="p"><</span><span class="nc">Properties</span><span class="p">,</span> <span class="nc">PropertiesValueSource</span><span class="p">.</span><span class="nc">Params</span><span class="p">></span> <span class="p">{</span>
<span class="kd">interface</span> <span class="nc">Params</span> <span class="p">:</span> <span class="nc">ValueSourceParameters</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">configFile</span><span class="p">:</span> <span class="nc">RegularFileProperty</span>
<span class="p">}</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">obtain</span><span class="p">():</span> <span class="nc">Properties</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">configFile</span> <span class="p">=</span> <span class="n">parameters</span><span class="p">.</span><span class="n">configFile</span><span class="p">.</span><span class="n">asFile</span><span class="p">.</span><span class="k">get</span><span class="p">()</span>
<span class="c1">// The remaining code of creating/loading a file can be left intact.</span>
<span class="kd">val</span> <span class="py">result</span> <span class="p">=</span> <span class="nc">Properties</span><span class="p">()</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">configFile</span><span class="p">.</span><span class="nf">exists</span><span class="p">())</span> <span class="p">{</span>
<span class="n">result</span><span class="p">[</span><span class="s">"some.property"</span><span class="p">]</span> <span class="p">=</span> <span class="s">"default value"</span>
<span class="n">configFile</span><span class="p">.</span><span class="nf">bufferedWriter</span><span class="p">().</span><span class="nf">use</span> <span class="p">{</span>
<span class="n">result</span><span class="p">.</span><span class="nf">store</span><span class="p">(</span><span class="n">it</span><span class="p">,</span> <span class="s">""</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">result</span>
<span class="p">}</span>
<span class="n">configFile</span><span class="p">.</span><span class="nf">bufferedReader</span><span class="p">().</span><span class="nf">use</span> <span class="p">{</span> <span class="n">result</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">it</span><span class="p">)</span> <span class="p">}</span>
<span class="k">return</span> <span class="n">result</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kd">val</span> <span class="py">propsProvider</span> <span class="p">=</span> <span class="n">providers</span><span class="p">.</span><span class="nf">of</span><span class="p">(</span><span class="nc">PropertiesValueSource</span><span class="o">::</span><span class="k">class</span><span class="p">)</span> <span class="p">{</span>
<span class="n">parameters</span><span class="p">.</span><span class="n">configFile</span> <span class="p">=</span> <span class="n">layout</span><span class="p">.</span><span class="n">projectDirectory</span><span class="p">.</span><span class="nf">file</span><span class="p">(</span><span class="s">"config.properties"</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">propsProvider</span><span class="p">.</span><span class="k">get</span><span class="p">().</span><span class="nf">getProperty</span><span class="p">(</span><span class="s">"should.register.task"</span><span class="p">).</span><span class="nf">toBoolean</span><span class="p">())</span> <span class="p">{</span>
<span class="n">tasks</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="s">"someTask"</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>File checks, reads, and writes made inside the ValueSource implementation aren’t recorded as configuration inputs. Instead, when checking the cache fingerprint, the whole ValueSource is recomputed, and the cache entry is only invalidated if the returned value changes.</p>
<p>Suppressing this type of input can be dangerous unless the user never modifies the configuration file.</p>
<h4 id="ensuring-the-file-exists-before-the-execution-phase">Ensuring the file exists before the execution phase</h4>
<p>Sometimes plugins or build logic create files at the configuration phase to make sure these files are available to the tasks running at the execution phase.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">someFile</span> <span class="p">=</span> <span class="nf">file</span><span class="p">(</span><span class="s">"createMe.txt"</span><span class="p">)</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">someFile</span><span class="p">.</span><span class="nf">exists</span><span class="p">())</span> <span class="p">{</span>
<span class="n">someFile</span><span class="p">.</span><span class="nf">writeText</span><span class="p">(</span><span class="s">"Some initial text"</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">abstract</span> <span class="kd">class</span> <span class="nc">Consumer</span> <span class="p">:</span> <span class="nc">DefaultTask</span><span class="p">()</span> <span class="p">{</span>
<span class="err">@</span><span class="k">get</span><span class="p">:</span><span class="nc">InputFile</span>
<span class="err">@</span><span class="k">get</span><span class="p">:</span><span class="nc">PathSensitive</span><span class="p">(</span><span class="nc">PathSensitivity</span><span class="p">.</span><span class="nc">NONE</span><span class="p">)</span>
<span class="k">abstract</span> <span class="kd">val</span> <span class="py">inputEnsured</span><span class="p">:</span> <span class="nc">RegularFileProperty</span>
<span class="nd">@TaskAction</span>
<span class="k">fun</span> <span class="nf">action</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">println</span><span class="p">(</span><span class="n">inputEnsured</span><span class="p">.</span><span class="n">asFile</span><span class="p">.</span><span class="k">get</span><span class="p">().</span><span class="nf">readText</span><span class="p">())</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">tasks</span><span class="p">.</span><span class="n">register</span><span class="p"><</span><span class="nc">Consumer</span><span class="p">>(</span><span class="s">"consumer"</span><span class="p">)</span> <span class="p">{</span>
<span class="n">inputEnsured</span> <span class="p">=</span> <span class="n">someFile</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Creating files at configuration time isn’t well supported with the configuration cache, especially when the file is then used at execution time. Gradle doesn’t track if files are created, deleted, or written at configuration time, so changes to such files may not invalidate the configuration cache. Creating the file with a task that other tasks depend on is a better solution.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">abstract</span> <span class="kd">class</span> <span class="nc">EnsureFile</span> <span class="p">:</span> <span class="nc">DefaultTask</span><span class="p">()</span> <span class="p">{</span>
<span class="err">@</span><span class="k">get</span><span class="p">:</span><span class="nc">OutputFile</span>
<span class="k">abstract</span> <span class="kd">val</span> <span class="py">ensured</span><span class="p">:</span> <span class="nc">RegularFileProperty</span>
<span class="nd">@TaskAction</span>
<span class="k">fun</span> <span class="nf">action</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">file</span> <span class="p">=</span> <span class="n">ensured</span><span class="p">.</span><span class="n">asFile</span><span class="p">.</span><span class="k">get</span><span class="p">()</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">file</span><span class="p">.</span><span class="nf">exists</span><span class="p">())</span> <span class="p">{</span>
<span class="n">file</span><span class="p">.</span><span class="nf">writeText</span><span class="p">(</span><span class="s">"Some initial text"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kd">val</span> <span class="py">ensureFile</span> <span class="k">by</span> <span class="n">tasks</span><span class="p">.</span><span class="nf">registering</span><span class="p">(</span><span class="nc">EnsureFile</span><span class="o">::</span><span class="k">class</span><span class="p">)</span> <span class="p">{</span>
<span class="n">ensured</span> <span class="p">=</span> <span class="n">layout</span><span class="p">.</span><span class="n">buildDirectory</span><span class="p">.</span><span class="nf">file</span><span class="p">(</span><span class="s">"createMeTask.txt"</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">tasks</span><span class="p">.</span><span class="n">register</span><span class="p"><</span><span class="nc">Consumer</span><span class="p">>(</span><span class="s">"consumer"</span><span class="p">)</span> <span class="p">{</span>
<span class="n">inputEnsured</span> <span class="p">=</span> <span class="n">ensureFile</span><span class="p">.</span><span class="nf">flatMap</span> <span class="p">{</span> <span class="n">it</span><span class="p">.</span><span class="n">ensured</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h4 id="using-files-for-cross-process-locks">Using files for cross-process locks</h4>
<p>Files can be used for cross-process lock protocols. For example, to ensure that concurrent builds do not overwrite some shared resource. When the locks are used at configuration time, a lock file can become an input to the configuration.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">inline</span> <span class="k">fun</span> <span class="nf">withFileLock</span><span class="p">(</span><span class="n">lockFile</span><span class="p">:</span> <span class="nc">File</span><span class="p">,</span> <span class="n">block</span><span class="p">:</span> <span class="p">()</span> <span class="p">-></span> <span class="nc">Unit</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">lockFile</span><span class="p">.</span><span class="nf">exists</span><span class="p">())</span> <span class="p">{</span>
<span class="n">lockFile</span><span class="p">.</span><span class="nf">createNewFile</span><span class="p">()</span>
<span class="p">}</span>
<span class="kd">val</span> <span class="py">channel</span> <span class="p">=</span> <span class="nc">FileChannel</span><span class="p">.</span><span class="k">open</span><span class="p">(</span><span class="n">lockFile</span><span class="p">.</span><span class="nf">toPath</span><span class="p">(),</span>
<span class="nc">StandardOpenOption</span><span class="p">.</span><span class="nc">READ</span><span class="p">,</span>
<span class="nc">StandardOpenOption</span><span class="p">.</span><span class="nc">WRITE</span><span class="p">,</span>
<span class="nc">StandardOpenOption</span><span class="p">.</span><span class="nc">CREATE</span>
<span class="p">)</span>
<span class="n">channel</span><span class="p">.</span><span class="nf">use</span> <span class="p">{</span>
<span class="n">channel</span><span class="p">.</span><span class="nf">tryLock</span><span class="p">()</span><span class="o">?.</span><span class="nf">use</span> <span class="p">{</span>
<span class="nf">block</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Both checking the file existence and opening the FileChannel for reading (by using <code class="language-plaintext highlighter-rouge">open(..., StandardOpenOption.READ, ...)</code>) make the lock file an input. The second run gets no cache hit because the result of the <code class="language-plaintext highlighter-rouge">exists</code> check changes.</p>
<p>When implementing the locking protocol, the content of the lock file typically doesn’t matter, so there is no need to open the file for reading. A race between checking for file existence and creating it is unavoidable if done separately (as shown above), so the check isn’t necessary either. After removing both the existence check and the <code class="language-plaintext highlighter-rouge">READ</code> option, the file is no longer an input:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">inline</span> <span class="k">fun</span> <span class="nf">withFileLockNoInput</span><span class="p">(</span><span class="n">lockFile</span><span class="p">:</span> <span class="nc">File</span><span class="p">,</span> <span class="n">block</span><span class="p">:</span> <span class="p">()</span> <span class="p">-></span> <span class="nc">Unit</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">channel</span> <span class="p">=</span> <span class="nc">FileChannel</span><span class="p">.</span><span class="k">open</span><span class="p">(</span><span class="n">lockFile</span><span class="p">.</span><span class="nf">toPath</span><span class="p">(),</span>
<span class="nc">StandardOpenOption</span><span class="p">.</span><span class="nc">WRITE</span><span class="p">,</span>
<span class="nc">StandardOpenOption</span><span class="p">.</span><span class="nc">CREATE</span>
<span class="p">)</span>
<span class="n">channel</span><span class="p">.</span><span class="nf">use</span> <span class="p">{</span>
<span class="n">channel</span><span class="p">.</span><span class="nf">tryLock</span><span class="p">()</span><span class="o">?.</span><span class="nf">use</span> <span class="p">{</span>
<span class="nf">block</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Suppressing this type of input is typically benign.</p>
<h4 id="file-access-in-third-party-libraries">File access in third-party libraries</h4>
<p>Third-party libraries used in plugin implementations may also access files. The libraries often do not depend on the Gradle API, which makes targeted fixes unlikely.</p>
<p>Plugin authors can either wrap uses of such libraries in ValueSources (if the library-provided value has to be obtained at configuration time) or use the <a href="https://docs.gradle.org/current/userguide/worker_api.html">Worker API</a> with classloader or process isolation to run the library code (if there’s no need in communicating back anything but success/failure). However, the latter is more suitable for task actions than the configuration phase.</p>
<h2 id="use-case-2-tracking-inputs-while-storing-the-task-graph">Use case 2: Tracking inputs while storing the task graph</h2>
<p>Another change to the build configuration input detection is planned for Gradle 8.4.</p>
<p>When storing the computed task graph in the cache, Gradle resolves configurations and flattens <a href="https://docs.gradle.org/current/userguide/lazy_configuration.html">provider</a> chains, replacing them with the computed values if possible. User code, particularly dependency resolution callbacks and <code class="language-plaintext highlighter-rouge">provider {}</code> calculations, runs when doing this. Before Gradle 8.4, inputs accessed while storing the computed task graph were ignored, which could lead to false cache hits.</p>
<p>For example, consider a task that only runs if the file is present:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tasks</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="s">"onlyIfFileExists"</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">fileExistsProvider</span> <span class="p">=</span> <span class="nf">provider</span> <span class="p">{</span> <span class="nf">file</span><span class="p">(</span><span class="s">"runTask.txt"</span><span class="p">).</span><span class="nf">exists</span><span class="p">()</span> <span class="p">}</span>
<span class="nf">onlyIf</span> <span class="p">{</span> <span class="n">fileExistsProvider</span><span class="p">.</span><span class="k">get</span><span class="p">()</span> <span class="p">}</span>
<span class="nf">doLast</span> <span class="p">{</span>
<span class="nf">println</span><span class="p">(</span><span class="s">"File exists!"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>When running on Gradle 8.3, the absence of the “runTask.txt” file is cached in the configuration cache, so even if the file is added later, the cache isn’t invalidated, and the task is still skipped. Gradle 8.4 fixes this, and the configuration cache is invalidated correctly when the file is added.</p>
<h3 id="temporarily-disabling-store-time-inputs">Temporarily disabling store-time inputs</h3>
<p>As with file API tracking, users may temporarily opt out of the new behavior by setting the Gradle property <code class="language-plaintext highlighter-rouge">org.gradle.configuration-cache.inputs.unsafe.ignore.in-serialization</code> to <code class="language-plaintext highlighter-rouge">true</code>. Note that there is no way to ignore only a subset of inputs accessed that way.</p>
<h3 id="fixing-plugins-or-build-logic-to-work-with-store-time-inputs">Fixing plugins or build logic to work with store-time inputs</h3>
<p>To avoid unwanted inputs at the store time, build and plugin authors can use the same techniques as the other configuration-time code. Another option is to replace the Callable-based provider with a ValueSource-based or built-in one, which may improve the configuration cache hit rate if the provider’s value is only queried at execution time.</p>
<p>The example above can be rewritten to avoid invalidating the configuration cache at all:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tasks</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="s">"onlyIfFileExists"</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">fileExistsProvider</span> <span class="p">=</span> <span class="n">providers</span><span class="p">.</span><span class="nf">fileContents</span><span class="p">(</span><span class="n">layout</span><span class="p">.</span><span class="n">projectDirectory</span><span class="p">.</span><span class="nf">file</span><span class="p">(</span><span class="s">"runTask.txt"</span><span class="p">)).</span><span class="n">asBytes</span>
<span class="c1">// fileContents provider has value present only if the file exists.</span>
<span class="nf">onlyIf</span> <span class="p">{</span> <span class="n">fileExistsProvider</span><span class="p">.</span><span class="n">isPresent</span> <span class="p">}</span>
<span class="nf">doLast</span> <span class="p">{</span>
<span class="nf">println</span><span class="p">(</span><span class="s">"File exists!"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>Detecting build configuration inputs is critical for the correctness of the builds that use the configuration cache. Gradle continues to improve in this area but allows existing builds to temporarily opt out from the new behavior if it incurs “benign” cache misses until build logic and plugins can be adequately fixed.</p>
<p>Let us know if you think Gradle doesn’t recognize a build configuration input or lacks APIs to support your use case in a configuration-cache compatible way by <a href="https://github.com/gradle/gradle/issues">opening a Gradle issue</a>. You can also contact us on <a href="https://discuss.gradle.org/">our forums</a> or in the <code class="language-plaintext highlighter-rouge">#configuration-cache</code> channel of the <a href="https://gradle.com/slack-invite">Gradle Community Slack</a>.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>For example, you can find the latest recommendations for the Android Gradle plugin in its <a href="https://developer.android.com/build/optimize-your-build#use-the-configuration-cache">documentation</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
An In-depth Look at Gradle's Approach to Faster Compilation2023-08-25T00:00:00-04:00https://blog.gradle.org/our-approach-to-faster-compilationOctavia Togami
<p>One of many performance optimizations that make Gradle Build Tool fast and scalable is compilation avoidance. Gradle avoids recompiling as much as possible by determining if the result of compilation would be identical, even if upstream dependencies have changed.</p>
<p>The situation can be illustrated like this: if classes <code class="language-plaintext highlighter-rouge">MyApp</code> and <code class="language-plaintext highlighter-rouge">NumberUtils</code> are in different projects and <code class="language-plaintext highlighter-rouge">MyApp</code>’s project depends on <code class="language-plaintext highlighter-rouge">NumberUtils</code>’s project at compile time, then any internal change to <code class="language-plaintext highlighter-rouge">NumberUtils</code> does not require <code class="language-plaintext highlighter-rouge">MyApp</code>’s project to be recompiled. Both before and after the change, <code class="language-plaintext highlighter-rouge">MyApp</code> compiles to identical bytecode, so Gradle can continue to use the <code class="language-plaintext highlighter-rouge">MyApp.class</code> file it has already built.</p>
<p><img src="/images/our-approach-to-faster-compilation/myapp-recompile-diff.png" alt="A GitHub Gist showing MyApp.java which uses NumberUtils.sum, and a NumberUtils.java.diff which shows a change from an imperative for-loop summation to a Stream-based reduce. On the right, it says "Does MyApp need to be recompiled? No!"" /></p>
<p>Gradle is smart enough to avoid recompiling a class if and only if two conditions are true: 1) its own code hasn’t changed and 2) any changes to classes it compiles against are ABI-compatible. As discussed previously in the <a href="https://blog.gradle.org/compilation-avoidance">Compilation Avoidance blog post</a>, this is a complementary feature to incremental compilation and works without the need to generate “header JARs” or “ABI JARs” known from some other build systems.</p>
<p>In this post, we compare these two approaches to compilation avoidance (with and without the generation of header JARs) and measure how skipping header JAR generation allows Gradle to achieve superior build performance.</p>
<h2 id="definitions">Definitions</h2>
<p>Before we dive into performance comparisons, let’s clarify a few important concepts.</p>
<h3 id="what-is-an-abi">What is an ABI?</h3>
<p>ABI stands for “Application Binary Interface”.</p>
<p>In Java, this means the parts of the library that are exposed to consumers, such as most public classes, public methods, and public fields. A library’s ABI does not include the bodies of methods, private classes, private methods, or private fields.</p>
<p>If a library changes its ABI, this can cause compilation failures or runtime errors for downstream consumers. If a library’s ABI has not changed between two versions, we call those versions “ABI-compatible” or “binary compatible”.</p>
<p>A full definition of binary compatibility can be found in the <a href="https://docs.oracle.com/javase/specs/jls/se19/html/jls-13.html">Java Language Specification, Chapter 13</a>.</p>
<h3 id="what-is-a-jar-file">What is a JAR file?</h3>
<p>A JAR file is a compressed archive that contains <code class="language-plaintext highlighter-rouge">.class</code> and resource files for a library using the same format as a ZIP file and a <code class="language-plaintext highlighter-rouge">.jar</code> extension.</p>
<p>JAR files are used by javac to compile Java source code against a library’s ABI.</p>
<h3 id="what-is-a-header-jar">What is a header JAR?</h3>
<p>A header JAR (also known as an “ABI JAR”) is created by taking the original set of classes or sources and removing the parts that are not part of the ABI. This means that the file contents of the header JAR itself represents the ABI of the library.</p>
<h3 id="what-is-compilation-avoidance">What is compilation avoidance?</h3>
<p>Compilation avoidance is an optimization done by build systems to avoid the expensive work of compiling if the output wouldn’t change. A compilation can be avoided if the ABI of the input classes remains the same.</p>
<p>This is safe because even if a library’s internals are changing, there is still no need to recompile unchanged downstream consumers unless its ABI changes. Internal changes to the implementation of a library may result in different behavior at <em>runtime</em> for the consumer, but cannot change the result of the consumer’s <em>compilation</em>.</p>
<h2 id="comparing-the-approaches">Comparing the approaches</h2>
<p>Some build systems, such as Bazel, generate header JARs in order to implement compilation avoidance. On the other hand, Gradle does compilation avoidance without generating header JARs. This section compares the two approaches.</p>
<h3 id="how-do-build-systems-that-use-header-jars-do-compilation-avoidance">How do build systems that use header JARs do compilation avoidance?</h3>
<p>Build systems typically rely on comparing the exact file contents between the previous and current build to determine if work should be done. If the file changes in any way, the old output is invalidated and must be built again. By using header JARs as the inputs to compile downstream projects (instead of the full JAR file containing implementation code like method bodies), ABI-compatible changes in dependencies will produce the same file contents for the header JAR. This allows the build system to use the file content to determine if a compilation should be avoided.</p>
<p>A downside of this approach is that there is no finer-grained level of avoidance – even ABI changes in internal library classes that are never used by a downstream project can cause recompilation because the contents of the header JAR will still change.</p>
<p>Additionally, as header JARs are typically not distributed, these build systems must generate them upon downloading a library JAR for the first time, and usually store both copies of the JAR.</p>
<p>For local dependencies, the header JAR can be generated in parallel with real compilation, which can allow downstream consumers to start compiling faster. In practice, we noticed that header JAR generation was much slower than compilation, resulting in a net penalty.</p>
<h3 id="how-does-gradle-do-compilation-avoidance-without-header-jars">How does Gradle do compilation avoidance without header JARs?</h3>
<p>Instead of only relying on file contents to detect changes, Gradle analyzes JARs and directories that are used on a <a href="https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/CompileClasspath.html">compile classpath</a>. This analysis is similar to the process of creating a header JAR, but instead of emitting a new JAR file, which would require lots of disk I/O and take extra space, Gradle directly checks the ABI against the previous ABI. In this way, Gradle still gets the benefits of a header JAR without the extra costs.</p>
<p>In addition, because Gradle has the full class analysis available, it also knows exactly which classes were changed, and can avoid recompiling any files that do not depend on them, avoiding even more compilation than header JARs.</p>
<p>Gradle also does not handle local dependencies any differently from remote dependencies, aside from using the results of compilation directly in class directories instead of packaging them into a JAR.</p>
<h3 id="process-details">Process Details</h3>
<p>Imagine you had a project that builds a Java application that has a Java library dependency<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>. A typical workflow is to edit a source file, then re-run the application. If the source file you edit is in the library, then the build system will need to perform the following steps:</p>
<p><img src="/images/our-approach-to-faster-compilation/build-steps-graph.png" alt="A graph showing the order of work done by Gradle and Bazel. It shows two execution columns, connected from left to right by dotted arrows. Dashes surround the application-related compilation and JAR steps." /></p>
<p>The dotted dependencies only exist for Gradle, as Bazel analyzes the source code for a local project dependency instead of using the output of the library compilation, and also compiles against the header JAR. This means that the two sides of this graph can be done fully in parallel for Bazel, only coming together when they are needed to run the application.</p>
<p>The dashed nodes are skipped if the new ABI is the same as the previous ABI.</p>
<p>Even though Bazel can do more work in parallel, due to the large amount of time it takes to write new class files, compress them into a new JAR, and write that to disk, it ends up being much slower due to the generation of the header JAR.</p>
<h2 id="performance-comparison">Performance comparison</h2>
<p>Using the <a href="https://github.com/gradle/gradle-profiler">Gradle Profiler</a>, we can capture build timing operation profiles from both Gradle and Bazel. See the <a href="https://github.com/gradle/Compilation-Avoidance-Experiment">experimental project used here</a> for instructions on how to reproduce these measurements.</p>
<p>We looked at two scenarios. One involves an ABI change and the other a non-ABI (internal) change. These scenarios are designed to highlight performance differences relevant to the usage of header JARs to avoid unnecessary re-compilation, not as a comprehensive comparison of Gradle and Bazel.</p>
<p>These measurements were performed using the latest currently available version of Bazel (6.2.1) and recent versions of Gradle (8.0+) on an Apple M2 Max MacBook Pro with 64 GB running macOS 13.4.1 using the Temurin 17.0.7 JDK. Where in particular Gradle is spending its build time can be discovered by setting the Gradle Profiler to produce low-level <a href="https://github.com/gradle/gradle-profiler#chrome-trace">Chrome traces</a> and interpreting the results.</p>
<h3 id="project-structure">Project structure</h3>
<p>Our experiment used a synthetic 1000 project build with 10 source files per project and inter-project dependencies. This is a project structure more typical of Bazel than Gradle projects. There are various tradeoffs to either build tool’s typical way of structuring large builds, such as the number of inter-project dependencies the build system must consider with a greater number of projects.</p>
<p>Unlike Gradle, Bazel does not perform incremental compilation at the project level. In this experiment only a fraction (25%) of the projects directly depend on the changing class. However the projects that do include such a dependency use <code class="language-plaintext highlighter-rouge">Production0</code> either directly or transitively in every class, to minimize Gradle’s incremental compilation benefits.</p>
<p>The class <code class="language-plaintext highlighter-rouge">Production0</code> defined in <code class="language-plaintext highlighter-rouge">project0</code> is used directly by a quarter of the projects in this build (every project with a number divisible by 4).</p>
<p>For example the class <code class="language-plaintext highlighter-rouge">Production44</code> in <code class="language-plaintext highlighter-rouge">project4</code> contains the code:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">private</span> <span class="nc">Production0</span> <span class="n">property0</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Production0</span><span class="o">();</span>
<span class="kd">public</span> <span class="nc">Production0</span> <span class="nf">getProperty0</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">property0</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">setProperty0</span><span class="o">(</span><span class="nc">Production0</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
<span class="n">property0</span> <span class="o">=</span> <span class="n">value</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="nc">String</span> <span class="n">property1</span> <span class="o">=</span> <span class="n">property0</span><span class="o">.</span><span class="na">getProperty1</span><span class="o">();</span>
</code></pre></div></div>
<p>The big picture of inter-project dependencies looks like this:</p>
<p><img src="/images/our-approach-to-faster-compilation/project-structure.png" alt="A graph of the project layout. project3 depends on project0-2, project7 depends on project4-6, project4 depends on project0, and project8 depends on project0. An arrow indicates that the class Production0 will be changed." /></p>
<p>This pattern of project and class dependencies repeats for higher numbered projects. The red projects (which have numbers divisible by 4) contain classes such as <code class="language-plaintext highlighter-rouge">Production44</code> that <em>must</em> be recompiled whenever there is an ABI change to <code class="language-plaintext highlighter-rouge">Production0</code>. Various other projects, marked in yellow in the diagram, only transitively depend on <code class="language-plaintext highlighter-rouge">project0</code>, should not need to be recompiled, as they do not use the changed <code class="language-plaintext highlighter-rouge">Production0</code> type in any way. Most projects, marked in white, have no dependency relationship on <code class="language-plaintext highlighter-rouge">project0</code>.</p>
<p>In contrast, when there is a non-ABI change, only <code class="language-plaintext highlighter-rouge">Production0</code> in <code class="language-plaintext highlighter-rouge">project0</code> should need to be recompiled. Other projects with a direct dependency such as <code class="language-plaintext highlighter-rouge">project4</code> are not affected as the ABI of <code class="language-plaintext highlighter-rouge">project0</code> remains the same, and so there is no need for their recompilation.</p>
<h3 id="results">Results</h3>
<link href="https://cdnjs.cloudflare.com/ajax/libs/c3/0.7.15/c3.min.css" rel="stylesheet" type="text/css" />
<style>
.c3 text { font-size: 1.5em; }
.c3-legend-item text { font-size: 1.2em; }
</style>
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.16.0/d3.min.js" charset="utf-8"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/c3/0.7.15/c3.min.js"></script>
<script type="text/javascript">
function chartPerformance(elementSelector, title, data, colors) {
c3.generate({
bindto: elementSelector,
data: {
rows: data,
type: 'bar',
x: 'Scenario',
labels: true
},
grid: {
focus: {
show: false
}
},
axis: {
x: { type: 'category' },
y: { label: 'duration (ms)' }
},
legend: { position: 'bottom' },
title: { text: title },
color: { pattern: colors },
tooltip: {
grouped: false,
horizontal: true,
format: {
title: function (d) { return null; },
value: function (value, ratio, id, index) { return value + ' ms'; }
}
}
});
}
</script>
<div id="synthetic-benchmark"></div>
<script>
var data = [
['Scenario', 'Bazel', 'Gradle 8.0', 'Gradle 8.1', 'Gradle 8.3'],
['ABI Change', 40563, 13160, 5382, 4557],
['Non-ABI Change', 1835, 2585, 972, 516]
];
chartPerformance('#synthetic-benchmark', 'Average Build Times', data, ['#0e6c37', '#02303a', '#0ba3cd', '#42c3c3']);
</script>
<hr />
<p>When comparing Bazel and Gradle, one thing immediately noticeable is that both Bazel and Gradle show dramatic improvements in build time when handling a non-ABI change vs. an ABI change. When dealing with an ABI change however, Bazel is <em>much slower</em> than Gradle. For a non-ABI change, Bazel is actually slightly <em>faster</em> than Gradle 8.0. However, the latest versions of Gradle are also faster than Bazel for a non-ABI change.</p>
<p>In the ABI-change scenario, Bazel is slower mainly because it needs to generate new header JARs for ABI changes. Since many projects directly depend on the project containing the ABI modification, when building with Bazel these projects also need their header JARs recompiled. This is because Bazel doesn’t have a way to know the ABI change didn’t affect them.</p>
<p>Gradle can extract the new ABI to verify there is no need for these recompilations <em>without</em> generating and packaging a header JAR, avoiding work that can be quite expensive in terms of overall build time. Gradle’s incremental compilation also improves performance during the ABI change scenario, though this is less impactful in this project.</p>
<p>The measurements also demonstrate significant improvements in the latest Gradle releases, which reflects the Gradle team’s commitment to improving performance and scalability. The specific optimizations that affect the measured scenarios are explained below.</p>
<h3 id="performance-improvements-in-gradle-81">Performance improvements in Gradle 8.1</h3>
<p>Due to the large number of subprojects involved, much of the Gradle build time is spent configuring the build and recalculating the <a href="https://docs.gradle.org/current/userguide/build_lifecycle.html#task_execution_graph_assembly">Directed Acyclic Graph of task dependencies</a> rather than task execution.</p>
<p>In Gradle 8.1, the <a href="https://docs.gradle.org/current/userguide/configuration_cache.html">Configuration Cache</a> <a href="https://docs.gradle.org/8.1/release-notes.html#promoted-features">became stable</a>. With configuration caching enabled, subsequent invocations of the same tasks skip the work of the configuration phase and get right to task execution, speeding up Gradle builds in both scenarios.</p>
<p>Configuration cache was enabled in the 8.1 and 8.3 tests. Using this feature is a great way to improve overall build times.</p>
<h3 id="further-performance-improvements-in-gradle-83">Further performance improvements in Gradle 8.3</h3>
<p>As we analyzed Gradle execution traces to understand the results of this experiment, we noticed that Gradle was spending disproportionately more time starting fresh compiler daemons on each build than compiling.</p>
<p>Starting in the 8.3 release, Gradle reuses Java compiler daemons by keeping them around between builds in order to avoid repaying their startup costs with each compilation. This can speed up some builds significantly and requires no additional configuration. With this change, Gradle 8.3 is even faster than Gradle 8.1 in both scenarios above.</p>
<p>The persistent compiler deamons optimization is <a href="https://docs.gradle.org/8.3/release-notes.html#faster-java-compilation">enabled by default</a> for macOS and Linux in Gradle 8.3. Windows support is coming in the 8.4 release.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Compilation avoidance is an important build performance optimization that speeds up builds by avoiding recompilation of downstream consumers when a library’s ABI has not changed.</p>
<p>In this blog post, we discussed two approaches to compilation avoidance: an approach relying on header JARs used in some build systems and the approach used by Gradle that calculates ABIs and short-circuits header JAR generation.</p>
<p>Gradle’s ability to analyze the classpath directly instead of building a header JAR for each change means it does not spend extra time creating and compiling partial class files and writing another JAR to disk. This advantage scales with the number of subprojects in a build.</p>
<p>For a large build with many interconnected projects that we measured, Gradle’s classpath analysis and incremental compilation already result in a much faster build for the ABI change scenario. For the non-ABI scenario, Gradle is faster with more recent optimizations like configuration cache (stable in Gradle 8.1) and persistent compiler daemons (enabled by default in Gradle 8.3).</p>
<p>Be sure to <a href="https://gradle.org/install/">upgrade soon</a> to take advantage of these improvements and more, and continue Building Happiness!</p>
<p>Thanks to <a href="https://github.com/tresat">Tom Tresansky</a>, who co-authored this blog post, and <a href="https://github.com/stepango">Stepan Goncharov</a>, who provided key insights about Bazel.</p>
<h2 id="notes">Notes</h2>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Assuming a non-JPMS build, which may affect certain details. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Simpler Kotlin DSL Property Assignment2023-06-30T00:00:00-04:00https://blog.gradle.org/simpler-kotlin-dsl-property-assignmentAnže Sodja
<p>One of the improvements that made <a href="https://blog.gradle.org/kotlin-dsl-is-now-the-default-for-new-gradle-builds">making Kotlin DSL the default for new Gradle builds</a> possible was a simpler way to assign values with <a href="https://docs.gradle.org/current/userguide/lazy_configuration.html#lazy_properties">lazy properties</a>.</p>
<p>In this blog post, we will explain what lazy property assignment is and how you can take advantage of the new syntax in the latest Gradle release.</p>
<h2 id="what-is-the-lazy-property-api">What is the lazy property API?</h2>
<p>The <a href="https://docs.gradle.org/current/userguide/lazy_configuration.html#lazy_properties">lazy property API</a> is a Gradle feature that delays the calculation of a property’s value until its required.
This is the recommended way of writing properties for extensions and tasks.</p>
<p>Some of the provided benefits include:</p>
<ul>
<li>Solving configuration ordering issues (no more <code class="language-plaintext highlighter-rouge">Project.afterEvaluate()</code>)</li>
<li>Tracking dependencies automatically (no need for <code class="language-plaintext highlighter-rouge">Task.dependsOn()</code>)</li>
<li>Providing standardized access (no custom behavior in setters or getters)</li>
<li>Reducing boilerplate for property declarations (no field and a setter, just a getter)</li>
</ul>
<p>To use the lazy property API in a custom task, you replace primitive types with a property type. e.g.,</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">MyTask</span> <span class="o">:</span> <span class="nc">DefaultTask</span><span class="o">()</span> <span class="o">{</span>
<span class="nd">@get</span><span class="o">:</span><span class="nc">Input</span>
<span class="kt">var</span> <span class="nl">myProperty:</span> <span class="nc">String</span><span class="o">?</span> <span class="o">=</span> <span class="kc">null</span>
<span class="nd">@TaskAction</span>
<span class="n">fun</span> <span class="nf">run</span><span class="o">()</span> <span class="o">{</span>
<span class="c1">// Skipped for brevity</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Becomes:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">MyTask</span> <span class="o">:</span> <span class="nc">DefaultTask</span><span class="o">()</span> <span class="o">{</span>
<span class="nd">@get</span><span class="o">:</span><span class="nc">Input</span>
<span class="kd">abstract</span> <span class="n">val</span> <span class="nl">myProperty:</span> <span class="nc">Property</span><span class="o"><</span><span class="nc">String</span><span class="o">></span>
<span class="nd">@TaskAction</span>
<span class="n">fun</span> <span class="nf">run</span><span class="o">()</span> <span class="o">{</span>
<span class="c1">// Skipped for brevity</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="here-comes-trouble">Here comes trouble!</h2>
<p>While lazy properties provide several benefits, they may make configuring properties with the Kotlin DSL a bit verbose.</p>
<p>In Groovy DSL, lazy properties can be assigned with a simple <code class="language-plaintext highlighter-rouge">=</code>:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tasks</span><span class="o">.</span><span class="na">register</span><span class="o">(</span><span class="s2">"taskName"</span><span class="o">,</span> <span class="n">MyTask</span><span class="o">)</span> <span class="o">{</span>
<span class="n">myProperty</span> <span class="o">=</span> <span class="s2">"Constant string value"</span>
<span class="o">}</span>
</code></pre></div></div>
<p>However, in Kotlin DSL, assigning a value to a property required the use of a verbose <strong>set(…)</strong> method:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tasks</span><span class="o">.</span><span class="na">register</span><span class="o"><</span><span class="nc">MyTask</span><span class="o">>(</span><span class="s">"taskName"</span><span class="o">)</span> <span class="o">{</span>
<span class="n">myProperty</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">"Constant string value"</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Users have clearly wanted a simpler syntax that looked more declarative. The request for a <a href="https://github.com/gradle/gradle/issues/9268">concise and statically typed <code class="language-plaintext highlighter-rouge">Property<T></code> assignment</a> is one of the most highly voted GitHub issues. As the Kotlin language did not support overloading the assign operator, there were not many good choices to improve this situation.</p>
<p>Because of the verbosity, some plugin authors invented custom setters.
These setters were implemented in many different ways, further degrading the user experience.</p>
<p>We didn’t want to force a more verbose DSL when using lazy properties or discourage the use of lazy properties in all plugins.</p>
<h2 id="an-answer-for-simple-assignment">An answer for simple assignment</h2>
<p>For a task or extension that uses a lazy property type (e.g. <code class="language-plaintext highlighter-rouge">Property</code>), Gradle needs a simple way to assign values.</p>
<p>As of Gradle 8.2, it is now possible to use the <code class="language-plaintext highlighter-rouge">=</code> operator to assign values to <code class="language-plaintext highlighter-rouge">Property</code> types as a direct alternative to the <code class="language-plaintext highlighter-rouge">set(T)</code> method in Kotlin DSL scripts.</p>
<p>The verbose option:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tasks</span><span class="o">.</span><span class="na">register</span><span class="o"><</span><span class="nc">MyTask</span><span class="o">>(</span><span class="s">"taskName"</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// Using the set() method call</span>
<span class="n">myProperty</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">"Constant string value"</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>becomes:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tasks</span><span class="o">.</span><span class="na">register</span><span class="o"><</span><span class="nc">MyTask</span><span class="o">>(</span><span class="s">"taskName"</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// Using lazy property assignment</span>
<span class="n">myProperty</span> <span class="o">=</span> <span class="s">"Constant string value"</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This syntax also supports assigning lazy values, like the <code class="language-plaintext highlighter-rouge">set(Provider<T>)</code> method:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tasks</span><span class="o">.</span><span class="na">register</span><span class="o"><</span><span class="nc">MyTask</span><span class="o">>(</span><span class="s">"taskName"</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// otherTask.map() returns lazy value `Provider<T>`</span>
<span class="n">myProperty</span> <span class="o">=</span> <span class="n">otherTask</span><span class="o">.</span><span class="na">map</span> <span class="o">{</span> <span class="n">it</span><span class="o">.</span><span class="na">otherProperty</span> <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Kotlin DSL now looks much better.
As a bonus, this makes the migration from Groovy DSL to Kotlin DSL easier as it reduces inconsistencies between the two DSLs.</p>
<p>Lazy property assignment works for getter-only properties of type <code class="language-plaintext highlighter-rouge">Property</code> or <code class="language-plaintext highlighter-rouge">ConfigurableFileCollection</code>.
We do <span style="text-decoration:underline;">not</span> recommended that plugin authors implement custom setters for properties with these types.</p>
<h2 id="a-joint-effort-with-jetbrains">A joint effort with JetBrains</h2>
<p>Implementing lazy property assignment would not have been possible without the close collaboration with the Kotlin team.</p>
<p>Together, we implemented a Kotlin compiler plugin that provides an option to overload the <code class="language-plaintext highlighter-rouge">=</code> assignment operator for final properties.
This is similar to what the Kotlin language supports with <a href="https://kotlinlang.org/docs/operator-overloading.html">other operators</a> like <code class="language-plaintext highlighter-rouge">+=</code>.
Since writing such a plugin required changes in the Kotlin compiler, we were not able to implement this change until now.</p>
<p>Internally, Gradle uses the new compiler plugin and provides methods that overload the <code class="language-plaintext highlighter-rouge">=</code> operator for lazy types:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">fun</span> <span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="nc">Property</span><span class="o"><</span><span class="no">T</span><span class="o">>.</span><span class="na">assign</span><span class="o">(</span><span class="nl">value:</span> <span class="no">T</span><span class="o">?)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">value</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Because the feature is written as a Kotlin compiler plugin, we also get all the benefits of the Kotlin language including static typing, discoverability, and great IDE integration.</p>
<p>We can also proudly say that this was the first externally contributed Kotlin language feature! The new plugin ships directly with Kotlin.</p>
<p>Thank you to the JetBrains team for their support and collaboration on this feature.</p>
<h2 id="try-it-out-in-gradle-82">Try it out in Gradle 8.2</h2>
<p>Gradle added lazy property assignment support in Gradle 8.1 as <a href="https://docs.gradle.org/8.1.1/release-notes.html#kotlin-assign">an opt-in feature</a>.
Gradle 8.2 enables this by default, so you can <em>simply</em> use it in your <strong>*.gradle.kts</strong> scripts.</p>
<p>It’s important to note that the feature is still marked as incubating. Changes to the feature are possible, although unlikely.
We don’t recommend using this feature in published plugins until it’s marked as stable (likely in Gradle 8.3).</p>
<p>Lazy property assignment is supported in IntelliJ 2022.3 (2023.1.1 or newer is recommended) and Android Studio Giraffe (or newer).</p>
<p>There are three known issues with IDE integration:</p>
<ul>
<li><a href="https://youtrack.jetbrains.com/issue/KT-56941">KT-56941</a></li>
<li><a href="https://youtrack.jetbrains.com/issue/KT-56221">KT-56221</a></li>
<li><a href="https://youtrack.jetbrains.com/issue/KTIJ-24390">KTIJ-24390</a></li>
</ul>
<p>KT-56941 and KT-56221 were fixed with IntelliJ 2023.1.1.</p>
<p>KTIJ-24390 will be fixed in a future release.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Lazy property assignment:</p>
<ul>
<li>Reduces the verbosity of Kotlin DSL</li>
<li>Standardizes task and extension property access</li>
<li>Simplifies usage of lazy property API</li>
<li>Helps users migrate from Groovy to Kotlin</li>
</ul>
<p>If you want to learn more about lazy property assignment with Kotlin, check out the Gradle <a href="https://docs.gradle.org/current/userguide/kotlin_dsl.html#kotdsl:assignment">Kotlin DSL primer</a>.</p>
<p>If you are migrating an existing project from Groovy DSL to Kotlin DSL, be sure to read the <a href="https://docs.gradle.org/current/userguide/migrating_from_groovy_to_kotlin_dsl.html">migration guide</a> and consider using the lazy property API.</p>
<p>Join the <a href="https://gradle.org/slack-invite">community chat</a> (#kotlin-dsl channel) and <a href="https://discuss.gradle.org">forum</a> to ask questions, share experiences, and contribute to the development of Kotlin DSL.
Your feedback and contributions are highly valued and essential for shaping the future of the Gradle Build Tool.</p>
Plugin Portal Outage Followup2023-06-28T00:00:00-04:00https://blog.gradle.org/plugin-portal-outage-followupSterling Greene
<p>On June 23rd, 2023, at 9:00 UTC, Gradle users started experiencing issues resolving artifacts from the <a href="https://plugins.gradle.org">Gradle Plugin Portal</a> because of changes to artifacts hosted on JCenter by JFrog. JCenter stopped serving files directly and redirected all requests to Maven Central. This was an unannounced change.</p>
<p>We contacted JFrog, and they replied it was a test and they would revert back to having JCenter serve artifacts.</p>
<p>Around 18:30 UTC, JCenter returned to normal behavior.</p>
<p>This post describes the effect this outage had on builds, ways to recover from similar outages, and what we will do to eliminate the dependency on JCenter in the future.</p>
<p>Users still directly using JCenter should also refer to <a href="/jcenter-shutdown">our original blog post</a> about the shutdown of JCenter on Gradle builds in general.</p>
<h2 id="effects-on-gradle-plugins-usage-in-builds">Effects on Gradle plugins usage in builds</h2>
<p>When JFrog stopped serving artifacts from JCenter, all requests were redirected to Maven Central.
This can impact builds in different ways.
In this blog post, we focus on the resolution of Gradle plugins from the Plugin Portal.</p>
<h3 id="background-information">Background information</h3>
<p>The Plugin Portal only hosts artifacts related to Gradle plugins.
The Plugin Portal redirects Gradle builds to JCenter to resolve transitive dependencies required by plugins.
Some of these transitive dependencies are hosted by JCenter and some are mirrored from Maven Central.</p>
<p>Additionally, some very old versions of Gradle plugins are also hosted on JCenter.
This is because they were historically hosted on JFrog’s Bintray service.
Following Bintray’s shutdown in May 2021, we removed the Plugin Portal’s integration with Bintray, but we kept redirecting to JCenter for those artifacts because JFrog committed to keeping JCenter read-only indefinitely.</p>
<p>When changes are made to JCenter, the Plugin Portal may no longer serve some artifacts or serve different artifacts.</p>
<h3 id="missing-artifacts">Missing artifacts</h3>
<p>Any artifacts only available on JCenter become unavailable when JCenter stops serving artifacts.</p>
<p>For example, a library published to JCenter 10 years ago and never published to Maven Central would no longer be available.</p>
<p>As a consequence, Gradle builds may fail when trying to resolve dependencies from the Plugin Portal.
This includes:</p>
<ol>
<li>Builds that rely on very old versions of plugins.
Those plugins may have been only hosted on JCenter, and that version of the plugin may no longer be available.</li>
<li>Builds that rely on plugins that transitively depend on other libraries hosted elsewhere.
Libraries that were only hosted by JCenter may no longer be available.</li>
</ol>
<h3 id="different-versions">Different versions</h3>
<p>Any plugin that relies on artifacts that are published to both JCenter and Maven Central may resolve to different versions when using dynamic versions.</p>
<p>For example, a library published to JCenter and later published to Maven Central only has the versions published to Maven Central available.</p>
<p>Gradle builds may not fail to resolve dependencies, but the build may resolve different versions.</p>
<h3 id="different-artifacts">Different artifacts</h3>
<p>Any plugin that relies on artifacts that are published to both JCenter and Maven Central may behave differently.</p>
<p>For example, a library published to JCenter and later published to Maven Central could have different dependencies, even for the same version.
A library may have been built for Java 6 on JCenter and Java 8 on Maven Central.
Signed artifacts may be affected and signed by different keys.</p>
<p>Gradle builds may not fail to resolve dependencies, but the behavior of those dependencies may have changed.</p>
<p>Builds using <a href="https://docs.gradle.org/current/userguide/dependency_verification.html">dependency verification</a> should fail with verification failures because of these differences.
These failures require manual intervention when the upstream repository is serving different artifacts.</p>
<h2 id="recovering-from-broken-builds">Recovering from broken builds</h2>
<p>Gradle does not have control over changes to JCenter. This kind of outage may occur again before we are able to migrate away from JCenter.</p>
<p>There are a few things you can do to recover:</p>
<ol>
<li>Identify builds that directly use JCenter for build logic and change them to use Maven Central.</li>
<li>Identify build logic dependencies that are being served by JCenter.</li>
</ol>
<p>If you have dependencies that come from JCenter:</p>
<ol>
<li>Update to latest plugin version, as it may no longer require JCenter only dependencies.</li>
<li>Remove unused direct dependencies.</li>
<li>Switch to updated coordinates.
Some dependencies may have migrated to new coordinates on Maven Central.</li>
<li>Migrate to alternative libraries.
Some dependencies may have alternatives that are hosted on Maven Central.</li>
<li>For transitive dependencies, exclude dependencies that are not necessary.</li>
</ol>
<p>Note that most of the above can also be applied to application dependencies that are JCenter only.</p>
<h2 id="what-gradle-is-doing">What Gradle is doing</h2>
<p>Gradle will be analyzing the most recent version of each plugin and determine which require a transitive dependency that is only available on JCenter.
Our goal is to migrate away from JCenter with as little disruption to the community as possible to prevent unexpected changes like this again.</p>
<p>Based on that information, Gradle will publish a follow-up blog post with next steps and reach out to affected plugin authors to see if they can publish a version of their plugin that only requires dependencies hosted on Maven Central.</p>
<p>Plugins that already fail to resolve with a working JCenter, that is, plugins that have dependencies hosted in another repository, will not be part of that analysis.</p>
<h2 id="feedback">Feedback</h2>
<p>Let us know if you have any questions on our <a href="https://discuss.gradle.org/">forums</a> or <a href="https://gradle.com/slack-invite">Gradle Community Slack</a>.</p>
Kotlin DSL is Now the Default for New Gradle Builds2023-04-13T00:00:00-04:00https://blog.gradle.org/kotlin-dsl-is-now-the-default-for-new-gradle-buildsPaul Merlin
<p>Kotlin DSL for Gradle was introduced in version 3.0 of the Gradle Build Tool in <a href="https://blog.gradle.org/kotlin-meets-gradle">August 2016</a> and <a href="https://blog.gradle.org/kotlin-dsl-1.0">released as 1.0 in Gradle 5.0</a>.
Since then, it’s been growing in popularity and has greatly improved the authoring experience of many Gradle builds.</p>
<p>Kotlin DSL is now the default choice for new Gradle builds.
This means that when creating a new project with Gradle, including in IntelliJ IDEA (starting with 2023.1) and Android Studio (starting with Giraffe), Kotlin DSL is the default option.
Support for Groovy DSL will continue for existing projects or those who prefer to use it.</p>
<p>In this post, we will explore the benefits of Kotlin DSL and why it is becoming the recommended option for new Gradle builds.
We will also discuss some of the improvements that are planned for the future to make Kotlin DSL even better.</p>
<p><img src="/images/kotlin-dsl-by-default/kotlin-dsl-default-banner.png" alt="Kotlin DSL by default" /></p>
<h2 id="about-kotlin-dsl">About Kotlin DSL</h2>
<p>Kotlin is a powerful language that offers many features that make it an excellent fit for creating a DSL.
Its static type system, concise and expressive syntax, and support for functional programming constructs make it easy to create readable and composable DSLs.</p>
<p>Kotlin DSL for Gradle takes advantage of the Kotlin language to enable full IDE assistance for build authoring in IntelliJ IDEA and Android Studio.
This includes auto-completion, smart content assist, quick access to documentation, navigation to source, and context-aware refactoring.
With Kotlin DSL, you can edit your build logic with the same editing experience you are used to when working with your production and test code.</p>
<p>Gradle combines Kotlin language features with its capabilities to provide an elegant and extensible declarative build language that enables expressing any build clearly and understandably.
Dynamically added elements of the build model, like project dependencies (when using <a href="https://docs.gradle.org/current/userguide/platforms.html#sub:central-declaration-of-dependencies">version catalog</a>) or subproject names in multi-project builds, benefit from static typing and IDE assistance.
Gradle plugins extend Kotlin DSL by contributing extensions and tasks made available statically in build scripts.</p>
<video autoplay="" muted="" preload="all" width="100%">
<source src="/images/kotlin-dsl-by-default/kotlin-dsl-ide.webm" type="video/webm" />
<source src="/images/kotlin-dsl-by-default/kotlin-dsl-ide.mp4" type="video/mp4" />
</video>
<h2 id="kotlin-dsl-as-default-for-new-builds">Kotlin DSL as default for new builds</h2>
<p>Thanks to the collaborative effort between JetBrains, Google, and Gradle, Kotlin DSL reached the next level of maturity and is now an even more efficient and user-friendly tool for developers.</p>
<p>Many usability improvements, including full support for version catalogs, and faster code analysis and auto-completion in IntelliJ IDEA and Android Studio were delivered.
There is also a brand-new <a href="https://docs.gradle.org/nightly/kotlin-dsl/">DSL reference</a>.</p>
<p>Starting with Gradle 8.2, creating new builds using <code class="language-plaintext highlighter-rouge">gradle init</code> defaults to generating Gradle builds using Kotlin DSL.
In IntelliJ IDEA 2023.1, creating a new Java application defaults to using Kotlin DSL.
All wizards and templates default to using Kotlin DSL in Android Studio Giraffe.</p>
<p>Reading the Gradle documentation, you’ll notice that the user manual and samples display Kotlin DSL first.
In addition, the documentation for the Kotlin Gradle Plugin and other Kotlin and JetBrains plugins is now 100% Kotlin.
The Android documentation, including the Android Gradle Plugin and other Google plugins, recipes, sample applications, and guides, now use Kotlin DSL.</p>
<p>As you can see, this change spans multiple releases of multiple tools.
See the related announcements from the <a href="https://blog.jetbrains.com/kotlin/2023/04/kotlin-dsl-is-the-default-for-new-gradle-builds/">JetBrains Blog</a> and <a href="https://android-developers.googleblog.com/2023/04/kotlin-dsl-is-now-default-for-new-gradle-builds.html">Google Blog</a>.</p>
<video autoplay="" muted="" preload="all" width="100%">
<source src="/images/kotlin-dsl-by-default/kotlin-dsl-reference.webm" type="video/webm" />
<source src="/images/kotlin-dsl-by-default/kotlin-dsl-reference.mp4" type="video/mp4" />
</video>
<h2 id="current-limitations-and-next-steps">Current limitations and next steps</h2>
<p>This milestone makes Kotlin DSL the default and recommended choice for new projects.
We also believe that Kotlin DSL is a better choice for most existing projects.
However, very large projects with complex build logic currently using Groovy DSL are advised to wait to migrate.
In particular, when changing shared build logic with Kotlin DSL, the script compilation performance is currently slower than with Groovy DSL.
Note that Kotlin script compilation outputs can be shared in a remote cache, so this caveat only applies to the person actually changing the shared build logic when using the <a href="https://docs.gradle.org/current/userguide/build_cache.html">build cache</a>.
While the script compilation performance overhead won’t be noticeable for new projects, it can be problematic for projects with a lot of existing build logic.</p>
<p>There is an ongoing effort to improve Kotlin DSL script compilation performance with continued collaboration between Gradle, JetBrains, and Google.
Our next goal is to make Kotlin DSL the recommended choice for large builds too.</p>
<p>You should expect more usability improvements in the upcoming Gradle and IDEs releases.
For example, you will soon be able to use the = operator to assign Gradle’s <a href="https://docs.gradle.org/current/userguide/lazy_configuration.html">lazy properties</a> and benefit from a streamlined DSL for Gradle’s domain object containers.</p>
<h2 id="learning-more">Learning more</h2>
<p>If you want to learn more about Gradle’s Kotlin DSL, check out the Gradle <a href="https://docs.gradle.org/current/userguide/kotlin_dsl.html">Kotlin DSL primer</a> and the updated <a href="https://docs.gradle.org/nightly/kotlin-dsl/">Kotlin DSL reference</a>.
If you are migrating an existing project from Groovy DSL to Kotlin DSL, be sure to read the <a href="https://docs.gradle.org/current/userguide/migrating_from_groovy_to_kotlin_dsl.html">migration guide</a>.</p>
<p>Join the <a href="https://gradle.org/slack-invite">community chat</a> (<code class="language-plaintext highlighter-rouge">#kotlin-dsl</code> channel) and <a href="https://discuss.gradle.org/c/help-discuss">forums</a> to ask questions, share experiences, and contribute to the development of Kotlin DSL.
Your feedback and contributions are highly valued and essential for shaping the future of the Gradle Build Tool.</p>
How Gradle Works Part 3 - Build Script2023-03-09T21:00:00-05:00https://blog.gradle.org/how-gradle-works-3Bo Zhang
<p>Previously on <em>How Gradle Works</em>:</p>
<ol>
<li><a href="./how-gradle-works-1">How Gradle Part 1 - Startup</a></li>
<li><a href="./how-gradle-works-2">How Gradle Part 2 - Inside The Daemon</a></li>
</ol>
<p>This is the third blog of the series <em>How Gradle Works</em>.
In this blog we’ll explain what happens during build script execution.</p>
<h2 id="kotlin--groovy-dsl">Kotlin & Groovy DSL</h2>
<p>If you are a Java developer, when you open any Gradle build script (for example <code class="language-plaintext highlighter-rouge">build.gradle.kts</code> or <code class="language-plaintext highlighter-rouge">build.gradle</code>), the first thing that might confuse you is the special syntax of curly braces:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// Kotlin DSL:
plugins {
id("some.plugin") version "0.0.1"
}
// or Groovy DSL:
plugins {
id "some.plugin" version "0.0.1"
}
</code></pre></div></div>
<p>What’s this? What happens when Gradle executes these kinds of scripts?</p>
<p>The short answer is: they are DSLs (<a href="https://en.wikipedia.org/wiki/Domain-specific_language">Domain Specific Language</a>) on top of the <a href="https://kotlinlang.org/">Kotlin</a> programming language for <code class="language-plaintext highlighter-rouge">.gradle.kts</code> files, or on top of the <a href="https://groovy-lang.org">Groovy</a> programming language for <code class="language-plaintext highlighter-rouge">.gradle</code> files.
These DSLs have some implicit rules, making them appear very confusing.</p>
<h3 id="implicit-rule-1-lambdaclosure">Implicit Rule 1: Lambda/Closure</h3>
<p>First of all, <code class="language-plaintext highlighter-rouge">{ ... }</code> is a special object in both Groovy and Kotlin.
This object is called a <a href="https://kotlinlang.org/docs/lambdas.html">lambda in Kotlin</a>, or a <a href="https://groovy-lang.org/closures.html">closure in Groovy</a>.
They’re similar to function objects in other programming language, like Java’s lambda, or JavaScript’s function object.</p>
<p>You can think of <code class="language-plaintext highlighter-rouge">plugins { ... }</code> as a method invocation in which a Kotlin lambda object or Groovy closure object is passed as the argument, because both Groovy and Kotlin allow you to omit parentheses:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>plugins(function() {
...
})
</code></pre></div></div>
<p>Also, there is one noteworthy DSL: in Kotlin/Groovy, if the last parameter of a function is a lambda/closure, it’s allowed to be put outside of parentheses.
For example, the following code snippet:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tasks.register("myTask") {
...
doLast {
...
}
}
</code></pre></div></div>
<p>is equivalent to:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tasks.register("myTask", function() {
...
doLast(function() {
...
})
})
</code></pre></div></div>
<p>The code inside the function may be executed immediately or later, depending on the implementation of the specific method.</p>
<h3 id="implicit-rule-2-chained-method-invocation">Implicit Rule 2: Chained Method Invocation</h3>
<p>In the <code class="language-plaintext highlighter-rouge">plugins { }</code> example above, the Kotlin version: <code class="language-plaintext highlighter-rouge">id("some.plugin") version "0.0.1"</code> and the Groovy version: <code class="language-plaintext highlighter-rouge">id "some.plugin" version "0.0.1"</code> are both equivalent to the chained method invocation <code class="language-plaintext highlighter-rouge">id("some.plugin").version("0.0.1")</code>.</p>
<p>Wait, why?</p>
<p>Because the <code class="language-plaintext highlighter-rouge">version</code> in <code class="language-plaintext highlighter-rouge">id("some.plugin") version "0.0.1"</code> is actually an <a href="https://kotlinlang.org/docs/functions.html#infix-notation">infix function</a> in Kotlin, which is defined <a href="https://github.com/gradle/gradle/blob/dee59dbbf4f2947466eaa79e27127f9879f2129f/subprojects/kotlin-dsl/src/main/kotlin/org/gradle/kotlin/dsl/PluginDependenciesSpecScope.kt#L55">here</a>,
and <code class="language-plaintext highlighter-rouge">id "some.plugin" version "0.0.1"</code> is <a href="http://docs.groovy-lang.org/docs/latest/html/documentation/core-domain-specific-languages.html#_command_chains">“command chains”</a> in Groovy.</p>
<p>We won’t explain all the implicit rules in Groovy and Kotlin DSLs (as that would require another full blog series :-P), you should simply understand that they are mapped to Gradle API methods in some ways.
See <a href="https://docs.gradle.org/current/userguide/kotlin_dsl.html">Kotlin DSL Primer</a> and <a href="https://docs.gradle.org/current/userguide/groovy_build_script_primer.html">Groovy DSL Primer</a>.</p>
<p>But what is <code class="language-plaintext highlighter-rouge">this</code> of the method invocation <code class="language-plaintext highlighter-rouge">id("some.plugin")</code>?</p>
<p>The code inside the function is executed against a <code class="language-plaintext highlighter-rouge">this</code> object, which is called <a href="https://kotlinlang.org/docs/lambdas.html">a “receiver” in Kotlin lambda</a>, or <a href="https://groovy-lang.org/closures.html#closure-owner">a “delegate” in Groovy closure</a>.
Gradle determines the correct <code class="language-plaintext highlighter-rouge">this</code> object and invokes the method against that <code class="language-plaintext highlighter-rouge">this</code> object.
In this example, the <code class="language-plaintext highlighter-rouge">this</code> object is of <a href="https://github.com/gradle/gradle/blob/e6e5bff27f2a68e81350008f7d6f86b33fccd8d8/subprojects/core-api/src/main/java/org/gradle/plugin/use/PluginDependenciesSpec.java#L98">type <code class="language-plaintext highlighter-rouge">PluginDependenciesSpec</code></a>.</p>
<h2 id="build-script-execution">Build Script Execution</h2>
<p>Once we have unveiled all the mechanics of the DSL, a Gradle build script is simply a few Gradle API calls built on top of the DSLs.
Like almost all other programming languages, Gradle executes the build script line by line, top to bottom.</p>
<p><img src="/images/how-gradle-works/dsl-gradle-api.png" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>Of course, the build script must be compiled into bytecode before being executed in the JVM.
Gradle does this transparently, giving the impression that the build script is being interpreted and executed.</p>
<h2 id="external-dependencies-in-build-script">External Dependencies in Build Script</h2>
<p>Consider the following build script:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// build.gradle.kts
import com.android.build.gradle.tasks.LintGlobalTask
plugins {
id("com.android.application") version "7.4.0"
}
tasks.withType<LintGlobalTask>().configureEach {
...
}
</code></pre></div></div>
<p>How can this build script be compiled without specifying the dependency of <code class="language-plaintext highlighter-rouge">com.android.build.gradle.tasks.LintGlobalTask</code>?
You may say, “doesn’t it come from the <code class="language-plaintext highlighter-rouge">plugins { }</code> block below?”</p>
<p>But remember, to execute the <code class="language-plaintext highlighter-rouge">plugins { }</code> block, the build script must be compiled first.
Now it’s a chicken-and-egg issue: to get the dependency of <code class="language-plaintext highlighter-rouge">LintGlobalTask</code>, we must compile and run the build script, but to compile the build script, we must get the dependency of <code class="language-plaintext highlighter-rouge">LintGlobalTask</code>.</p>
<p>Gradle handles <code class="language-plaintext highlighter-rouge">plugins { }</code> specially as follows:</p>
<ol>
<li>The <code class="language-plaintext highlighter-rouge">plugins</code> block is extracted and executed first;</li>
<li>The resolved dependencies are added to the whole build script’s classpath;</li>
<li>The build script is compiled and executed.</li>
</ol>
<p><img src="/images/how-gradle-works/compile-plugins-first.png" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>A similar thing happens to <code class="language-plaintext highlighter-rouge">buildscript { }</code> block, too: you can explicitly specify the dependencies for build script compilation and execution.
In this way, you can leverage the many available libraries in the JVM ecosystem to empower your build script.</p>
<h2 id="whats-next">What’s Next</h2>
<p>The build script invokes Gradle APIs to configure the build, where amazing things happen.
It’s possible to pack the build script into a Gradle plugin for better reusability and performance.</p>
<p>In the next blog of the series, we’ll explain what happens under the hood of Gradle plugins.</p>
Gradle Wrapper Attack Report2023-01-25T00:00:00-05:00https://blog.gradle.org/wrapper-attack-reportLouis Jacomet
<p>On January 11th 2023, we were contacted by <a href="https://minecraftonline.com/">MinecraftOnline</a> about two unusual and suspicious Gradle wrapper JARs found in some of their repositories.
The wrappers were updated by a new contributor to MinecraftOnline.</p>
<p>We’ve performed an analysis of the JARs and will describe our findings below.
We have determined that one exploit was especially crafted as an attack against the MinecraftOnline project.</p>
<p>If you are not interested in all of the details, jump immediately to our <a href="/project-integrity">companion blog</a> covering how to protect your project or you, as a developer, against similar attacks.</p>
<h2 id="analysis">Analysis</h2>
<p>Our analysis started by confirming that the SHA256 checksums for both JARs did not match any of the <a href="https://gradle.org/release-checksums/">known good Gradle Wrapper checksums</a>:</p>
<ul>
<li>First JAR: <code class="language-plaintext highlighter-rouge">8449b6955690ec956c8ecfe1ae01e10a2aa76ddf18969985c070e345605acce1</code></li>
<li>Second JAR: <code class="language-plaintext highlighter-rouge">8e129181710bdc045423ddde59244586d7acbc0b2c5e2ddfc098559da559cf85</code></li>
</ul>
<p>After decompiling the two JARs, we discovered two exploits had been patched into the wrapper JAR.</p>
<h3 id="discord-credentials-stealing">Discord credentials stealing</h3>
<p>The first exploit, present in both JARs, attempts to steal <a href="https://discord.com">Discord</a> credentials by looking into specific files on the host computer.</p>
<p>The code is very similar to Discord token logging found online.
The exploit hides in different Gradle Wrapper classes and obfuscates <code class="language-plaintext highlighter-rouge">String</code> constants through a character array lookup.</p>
<p>Using a regular expression, lines from certain files are uploaded to a <a href="https://discord.com/developers/docs/resources/webhook#execute-webhook">Discord Webhook</a> using a hardcoded token found in the code.</p>
<h3 id="downloading-and-running-code-locally">Downloading and running code locally</h3>
<p>The second JAR contains an additional exploit.
On certain Gradle invocations, it will attempt to download another malicious JAR and then run it.</p>
<p>For this code path to trigger, the Gradle invocation needed to start with <code class="language-plaintext highlighter-rouge">publish</code> or <code class="language-plaintext highlighter-rouge">magic</code>.
<code class="language-plaintext highlighter-rouge">publish</code> is a Gradle task for pushing all project artifacts to a repository.
Builds that publish artifacts typically have access to higher privileged credentials.
We think that <code class="language-plaintext highlighter-rouge">magic</code> was used as a way to test the exploit.</p>
<p>Running that JAR resulted in the following actions:</p>
<ol>
<li>Edit the <code class="language-plaintext highlighter-rouge">build.gradle</code> file to modify the software being built by adding additional dependencies
<ol>
<li>Add two repositories, first in the list: a file-based one and <code class="language-plaintext highlighter-rouge">mavenCentral()</code></li>
<li>Add dependencies to the <code class="language-plaintext highlighter-rouge">shadow</code> configuration: the downloaded JAR itself and two third party libraries</li>
<li>Relocate the injected code to be in the <code class="language-plaintext highlighter-rouge">org.mariadb.jdbc.internal.cachevalidator</code> package as part of the <a href="https://imperceptiblethoughts.com/shadow/introduction/">Shadow plugin</a> configuration</li>
</ol>
</li>
<li>Modify a project specific source file so that the software would execute the malicious code
<ol>
<li>Add initialization of the code from the injected dependency</li>
</ol>
</li>
</ol>
<p>In addition to the above, the exploit would modify any Gradle invocation that started with <code class="language-plaintext highlighter-rouge">wrapper</code> to invoke <code class="language-plaintext highlighter-rouge">cleanEclipse</code> instead.
We think this was an attempt to make it harder for the malicious wrapper JAR to be removed.</p>
<h3 id="modified-files-in-the-wrapper-jars">Modified files in the wrapper JARs</h3>
<p>For completeness, here are the modified files found in the infected wrapper JARs:</p>
<ul>
<li>First infected wrapper
<ul>
<li><code class="language-plaintext highlighter-rouge">org/gradle/cli/SystemPropertiesCommandLineConverter.class</code>
<ul>
<li>File on <a href="https://www.virustotal.com/gui/file/aebb9a9ed3da2cc639df2fe097bd716d74e3bf68b7a3d890d0a1d62dae27e676/detection">VirusTotal</a></li>
</ul>
</li>
<li><code class="language-plaintext highlighter-rouge">org/gradle/wrapper/Download.class</code>
<ul>
<li>File on <a href="https://www.virustotal.com/gui/file/106e966c3d4f2e799a00b972354ee5df033adcd6e4f5d7fadd07559eefd3be6e?nocache=1">VirusTotal</a></li>
</ul>
</li>
<li><code class="language-plaintext highlighter-rouge">org/gradle/wrapper/PathAssembler.class</code>
<ul>
<li>File on <a href="https://www.virustotal.com/gui/file/b1b717c1e861a37d01763d62a886260d93083df4dfbaecb49cff7a0478ebed87?nocache=1">VirusTotal</a></li>
</ul>
</li>
</ul>
</li>
<li>Second infected wrapper
<ul>
<li><code class="language-plaintext highlighter-rouge">org/gradle/cli/CommandLineParser.class</code>
<ul>
<li>File on <a href="https://www.virustotal.com/gui/file/df17c7f3689b472d77e065f007f3da816e11c408d8a175e731ebcdd2b5871f97/">VirusTotal</a></li>
</ul>
</li>
<li><code class="language-plaintext highlighter-rouge">org/gradle/cli/SystemPropertiesCommandLineConverter.class</code>
<ul>
<li>Same as in the first wrapper</li>
</ul>
</li>
<li><code class="language-plaintext highlighter-rouge">org/gradle/wrapper/Download.class</code>
<ul>
<li>File on <a href="https://www.virustotal.com/gui/file/e63e69972db7fdddf1ab5447d8cc55eb6013e6773210fe94d966e44b7f25829d?nocache=1">VirusTotal</a></li>
</ul>
</li>
<li><code class="language-plaintext highlighter-rouge">org/gradle/wrapper/PathAssembler.class</code>
<ul>
<li>Same as in the first wrapper</li>
</ul>
</li>
</ul>
</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>While the Gradle team is aware of the potential attack vector of injecting a malicious wrapper JAR, this is the first report we received of it being actively exploited as a <a href="https://en.wikipedia.org/wiki/Supply_chain_attack">supply chain attack</a>.</p>
<p>In general, we advise caution when integrating any changes from untrusted sources that may affect your build process. Please report any suspicious projects, wrappers, or distributions to <a href="mailto:security@gradle.com">us</a>.
See the <a href="/project-integrity">companion blog post</a> about how to protect your project or you, as a developer, against similar attacks.</p>
<p>Let us know if you have any questions on our <a href="https://discuss.gradle.org/">forums</a> or <a href="https://gradle.com/slack-invite">Gradle Community Slack</a>, or start a discussion below.</p>
Protecting Project Integrity2023-01-25T00:00:00-05:00https://blog.gradle.org/project-integrityLouis Jacomet
<p>Our recent <a href="/wrapper-attack-report">security report</a> shows that <a href="https://en.wikipedia.org/wiki/Supply_chain_attack">supply chain attacks</a> targeting the build process through the Gradle Wrapper exist in the wild.
This blog post explains how to protect your project or you, as a developer, against similar attacks.</p>
<p>A build process, by design, executes code.
The components of the build process all carry their own risks:</p>
<ol>
<li>The bootstrapping script to run the build tool could be compromised. (see <a href="#how-to-ensure-gradle-wrapper-integrity">How to ensure Gradle wrapper integrity?</a>)</li>
<li>The build tool itself could be compromised. (see <a href="#how-to-ensure-gradle-distribution-integrity">How to ensure Gradle distribution integrity?</a>)</li>
<li>The build tool could download third-party dependencies that are themselves compromised. (see <a href="#how-to-ensure-third-party-dependencies-integrity">How to ensure third party dependencies’ integrity?</a>)</li>
<li>Malicious code could be hidden in the project code, tests or build configuration. (see <a href="#how-to-ensure-project-integrity">How to ensure project integrity?</a>)</li>
</ol>
<p>Because of all those possible attack vectors, you should exercise caution <em>when you are not fully confident in the source of the changes</em>.
In the case of the attack against <a href="https://minecraftonline.com/">MinecraftOnline</a> analyzed in <a href="/wrapper-attack-report">our report</a>, Gradle wrapper updates were done by a new contributor.</p>
<p>Similarly, contributing to open source potentially exposes you to such attacks.
It would be best to validate new projects you are looking into similarly.</p>
<p>The key point is that <strong><em>you should only run a build if you trust the project or change set</em></strong>.</p>
<p>The following recommendations can help establish trust in a local Gradle build.</p>
<h2 id="how-to-ensure-gradle-wrapper-integrity">How to ensure Gradle wrapper integrity?</h2>
<p>Firstly, ensure the integrity of Gradle Wrapper on each change of the wrapper JAR, especially coming from external contributors.
Gradle <a href="https://gradle.org/release-checksums/">publishes the checksums</a> of the <code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code> for each Gradle version.</p>
<h3 id="wrapper-integrity-for-maintainers">Wrapper integrity for maintainers</h3>
<ul>
<li>Gradle wrapper JARs should always be validated when updated in a repository
<ul>
<li>For those using GitHub actions, Gradle publishes a <a href="https://github.com/marketplace/actions/gradle-wrapper-validation">dedicated action</a> that will validate wrapper checksums</li>
<li>For any other setup, automating that verification can be done by expanding on our <a href="https://docs.gradle.org/current/userguide/gradle_wrapper.html#manually_verifying_the_gradle_wrapper_jar">validation instructions</a></li>
</ul>
</li>
<li>Prefer to update the wrapper yourself instead of merging a PR from an external contributor.
Regenerate the <code class="language-plaintext highlighter-rouge">gradlew</code>, <code class="language-plaintext highlighter-rouge">gradle-wrapper.properties</code>, and <code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code> from a known good Gradle distribution.
Invoking a local Gradle with <code class="language-plaintext highlighter-rouge">gradle wrapper</code> will not run the <code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code> of the project <em>but it will configure the build</em>.</li>
</ul>
<h3 id="wrapper-integrity-for-contributors">Wrapper integrity for contributors</h3>
<ul>
<li>If you do not trust the project you are building, prefer using a known good, local Gradle distribution over a wrapper.</li>
</ul>
<h2 id="how-to-ensure-gradle-distribution-integrity">How to ensure Gradle distribution integrity?</h2>
<p>Secondly, ensure the integrity of the Gradle distribution itself.
Alongside the wrapper checksums, you can also find the <a href="https://gradle.org/release-checksums/">distribution checksums</a>.</p>
<h3 id="distribution-integrity-for-maintainers">Distribution integrity for maintainers</h3>
<ul>
<li>Gradle distribution checksum can be <a href="https://docs.gradle.org/current/userguide/gradle_wrapper.html#sec:verification">added to the <code class="language-plaintext highlighter-rouge">gradle-wrapper.properties</code> file</a> to enforce that what is downloaded matches the expected checksum.</li>
</ul>
<h3 id="distribution-integrity-for-contributors">Distribution integrity for contributors</h3>
<ul>
<li>Here again, relying on a known good, local Gradle distribution can be safer.</li>
</ul>
<h2 id="how-to-ensure-third-party-dependencies-integrity">How to ensure third party dependencies’ integrity?</h2>
<p>Ensuring the integrity of third-party dependencies involves evaluating multiple factors, such as the familiarity and reliability of the dependencies or plugins being used, as well as the trustworthiness of the repositories they originate from.
Considering all these factors together is essential to properly verify their integrity.</p>
<h3 id="dependencies-integrity-for-maintainers">Dependencies integrity for maintainers</h3>
<ul>
<li>Gradle allows you to enable <a href="https://docs.gradle.org/current/userguide/dependency_verification.html">dependency verification</a>.
This feature ensures that third party dependencies, both build plugins and project dependencies, match the trusted signature or checksums that are configured.</li>
<li>Be wary of any changes to build scripts that use unusual repositories, plugins, or dependencies.</li>
</ul>
<h3 id="dependencies-integrity-for-contributors">Dependencies integrity for contributors</h3>
<ul>
<li>Be wary of any build scripts that use unusual repositories, plugins, or dependencies.
Dependency verification is not a guarantee of trustworthiness, it only ensures that what is used by the build matches expectations.</li>
</ul>
<h2 id="how-to-ensure-project-integrity">How to ensure project integrity?</h2>
<p>This is potentially a major effort, and the level of scrutiny will depend on many factors.
However, it’s important to understand that any time you run someone else’s code you are exposed to a potential security risk.</p>
<h3 id="project-integrity-for-maintainers-and-contributors">Project integrity for maintainers and contributors</h3>
<ul>
<li>Inspect the build scripts and any contributed code before running any tasks or importing a project into the IDE.
Even running <code class="language-plaintext highlighter-rouge">gradle help</code> could do something malicious.</li>
<li>Consider using throw away environments when running untrusted code, both locally and in a continuous integration environment.</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>Supply chain attacks are a sad reality as we showed in our recent <a href="/wrapper-attack-report">security report</a>.</p>
<p>The Gradle team will continue to look for ways to improve the security of the Gradle Build Tool and its community.
Please report any suspicious projects, wrappers, or distributions to <a href="mailto:security@gradle.com">us</a>.
However, in the end, security is the responsibility of every project and developer.
You should assess your risks, and needs.
And then act accordingly.</p>
<p>Let us know if you have any questions on our <a href="https://discuss.gradle.org/">forums</a> or <a href="https://gradle.com/slack-invite">Gradle Community Slack</a>, or start a discussion below.</p>
How Gradle Works Part 2 - Inside The Daemon2023-01-21T02:00:00-05:00https://blog.gradle.org/how-gradle-works-2Bo Zhang
<p>Previously on <em>How Gradle Works</em>:</p>
<ol>
<li><a href="./how-gradle-works-1">How Gradle Part 1 - Startup</a></li>
</ol>
<p>This is the second blog of the series <em>How Gradle Works</em>.
In this blog we’ll explain what happens inside the <code class="language-plaintext highlighter-rouge">Gradle Daemon JVM</code>.</p>
<h2 id="why-we-need-gradle-daemon">Why We Need Gradle Daemon?</h2>
<p>In <a href="https://blog.gradle.org/how-gradle-works-1">the last blog</a>, we mentioned that Gradle starts a <code class="language-plaintext highlighter-rouge">Gradle Daemon JVM</code> (“the daemon”) to run the build.
<a href="https://docs.gradle.org/current/userguide/gradle_daemon.html">The userguide</a> explains why we need the daemon.</p>
<blockquote>
<p>The Daemon is a long-lived background process that reduces the time it takes to run a build. The Daemon reduces build times by:</p>
<ul>
<li>
<p>caching project information across builds</p>
</li>
<li>
<p>running in the background so every Gradle build doesn’t have to wait for JVM startup</p>
</li>
<li>
<p>benefiting from continuous runtime optimization in the JVM</p>
</li>
<li>
<p>watching the file system to calculate exactly what needs to be rebuilt before you run a build</p>
</li>
</ul>
</blockquote>
<p>The Gradle daemon was introduced in Gradle 3.0 and became mature over the years.
It’s enabled by default, and we don’t recommend disabling it under any circumstances.</p>
<h2 id="what-happens-in-the-daemon">What Happens in the Daemon?</h2>
<p>After <code class="language-plaintext highlighter-rouge">Gradle Client JVM</code> (“the client”) connects to a compatible idle daemon, it sends the necessary build information (command line arguments, project directory, env variables, etc.) to the daemon.
The daemon then starts running the build and sends build output (logging, stdout/stderr, etc.) back to the client.
The communication happens via a local socket connection.</p>
<p><img src="/images/how-gradle-works/client-connects-to-daemon.png" width="70%" height="70%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>But what exactly happens inside the daemon?
<a href="https://docs.gradle.org/current/userguide/build_lifecycle.html">The userguide</a> explains that there are three phases in a Gradle build: <code class="language-plaintext highlighter-rouge">Initialization</code>, <code class="language-plaintext highlighter-rouge">Configuration</code>, and <code class="language-plaintext highlighter-rouge">Execution</code>.</p>
<h3 id="initialization-phase-creation-of-build-objects">Initialization Phase: Creation of Build Objects</h3>
<p>Now that the daemon knows everything about the build, it starts creating the internal representations for the build.
Because Gradle runs on the JVM, these representations are Java objects.</p>
<p>For example, the whole Gradle build invocation is represented by a <a href="https://github.com/gradle/gradle/blob/ba32027bf0656be5c8a71e6281939ff410a9cf1a/subprojects/core-api/src/main/java/org/gradle/api/invocation/Gradle.java">Gradle</a> instance.
The configuration required to configure the project hierarchy is represented by a <a href="https://github.com/gradle/gradle/blob/fd341b1e7016ff0ba82995b4e3211fb6e6805dd4/subprojects/core-api/src/main/java/org/gradle/api/initialization/Settings.java">Settings</a> instance.
There is a <a href="https://github.com/gradle/gradle/blob/6121fa83ce4ac07a27ee043d8e69b0f5f99d1c49/subprojects/core-api/src/main/java/org/gradle/api/Project.java">Project</a> instance corresponding to each project we’re trying to build.</p>
<p><img src="/images/how-gradle-works/creating-build-instances.png" width="70%" height="70%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p><code class="language-plaintext highlighter-rouge">Gradle</code>, <code class="language-plaintext highlighter-rouge">Settings</code> and <code class="language-plaintext highlighter-rouge">Project</code> are also the default delegation of init, settings and build script.
This means these objects can be later interacted with in the build script. For example, when we say <code class="language-plaintext highlighter-rouge">println(name)</code> in a build script,
we are actually calling <a href="https://github.com/gradle/gradle/blob/6121fa83ce4ac07a27ee043d8e69b0f5f99d1c49/subprojects/core-api/src/main/java/org/gradle/api/Project.java#L311"><code class="language-plaintext highlighter-rouge">Project.getName()</code> method</a> on the <code class="language-plaintext highlighter-rouge">Project</code> instance.</p>
<h3 id="configuration-phase-build-script-execution">Configuration Phase: Build Script Execution</h3>
<p>After the necessary JVM objects are created, Gradle will load and execute the build scripts in the daemon.
A build script is usually named <code class="language-plaintext highlighter-rouge">X.gradle</code> (Groovy DSL) or <code class="language-plaintext highlighter-rouge">X.gradle.kts</code> (Kotlin DSL) in the project directory.
Groovy and Kotlin are both JVM languages, meaning they can run seamlessly inside a JVM (i.e., the daemon JVM).</p>
<p>For example, the following Groovy build script creates <a href="https://groovy-lang.org/closures.html">a Groovy <code class="language-plaintext highlighter-rouge">Closure</code></a> instance and passes the <code class="language-plaintext highlighter-rouge">Closure</code> instance to <a href="https://github.com/gradle/gradle/blob/6121fa83ce4ac07a27ee043d8e69b0f5f99d1c49/subprojects/core-api/src/main/java/org/gradle/api/Project.java#L1533"><code class="language-plaintext highlighter-rouge">Project.repositories(Closure)</code> method</a> on the <code class="language-plaintext highlighter-rouge">Project</code> instance created in the previous initialization phase.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>repositories {
mavenCentral()
}
</code></pre></div></div>
<p>Don’t worry if you don’t fully understand build script execution at this stage.
We’ll explain the details of build script execution in the next blog of this series.
For now, we can simply understand Gradle as <a href="https://en.wikipedia.org/wiki/Interpreter_(computing)">an interpreter</a> that executes the build script line by line, top to bottom.</p>
<p><img src="/images/how-gradle-works/configuration-build-script-execution.png" width="80%" height="80%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>The build script populates the data structures for the build inside the daemon JVM.
For example, the following build script snippet registers a <code class="language-plaintext highlighter-rouge">hello</code> task into Gradle’s task container data structure (i.e., <a href="https://github.com/gradle/gradle/blob/632c55abe090f3c453b8e6220b21adfeb1062180/subprojects/core-api/src/main/java/org/gradle/api/tasks/TaskContainer.java">class <code class="language-plaintext highlighter-rouge">TaskContainer</code></a>), which means there will be a <a href="https://github.com/gradle/gradle/blob/d0d40ba69f4b8f24b36f6aaa5b51678e2272f792/subprojects/core-api/src/main/java/org/gradle/api/Task.java"><code class="language-plaintext highlighter-rouge">Task</code> instance</a> created when needed.
This process is usually called “configuration,” i.e., configuring the data structure; that’s why this phase is called the “configuration phase.”</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tasks.register("hello") {
doLast {
println("Hello world!")
}
}
</code></pre></div></div>
<p>After the build script execution finishes, the build data structures are configured with the necessary data for the build.
Now we are ready for the next phase: select some tasks and execute them.</p>
<h3 id="execution-phase-execution-of-selected-tasks">Execution Phase: Execution of Selected Tasks</h3>
<p>After the configuration phase, Gradle has all the necessary data for the build stored in the daemon JVM.
Then, it determines the subset of tasks to be executed by the arguments passed to the <code class="language-plaintext highlighter-rouge">gradle</code> command
and executes each of the selected tasks.</p>
<p><img src="/images/how-gradle-works/daemon-execution-phase.png" width="80%" height="80%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>Each <code class="language-plaintext highlighter-rouge">Task</code> has a list of actions made up of chunks of code to be executed.
For example, if you wonder what the <a href="https://github.com/gradle/gradle/blob/a45bfed1cd64efe32d8ca4f4414250247d5b3738/subprojects/testing-jvm/src/main/java/org/gradle/api/tasks/testing/Test.java"><code class="language-plaintext highlighter-rouge">Test</code> task</a> does,
just search <code class="language-plaintext highlighter-rouge">@TaskAction</code> in the source code, you’ll find:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class Test {
...
@TaskAction
public void executeTests() {
...
}
}
</code></pre></div></div>
<p>When we say “a task is executed,” we mean “the code in its actions is executed in the daemon JVM.”
The task actions are always executed in the daemon JVM, but the actions can decide to fork some new JVMs and run some code in the forked JVMs.</p>
<p>For example, the <a href="https://docs.gradle.org/current/userguide/worker_api.html">Gradle Worker API</a> provides a way to
break up the execution of a task action into pieces and then execute them in child processes.</p>
<p>Another example of custom task actions forking extra JVMs is
<a href="https://github.com/gradle/gradle/blob/a45bfed1cd64efe32d8ca4f4414250247d5b3738/subprojects/testing-jvm/src/main/java/org/gradle/api/tasks/testing/Test.java"><code class="language-plaintext highlighter-rouge">Test</code> task</a>.
The <code class="language-plaintext highlighter-rouge">Test</code> task action is executed in the daemon JVM but during the execution,
it forks a few JVMs and runs the test code inside the forked JVMs to avoid the test code interfering with the daemon JVM.</p>
<p><img src="/images/how-gradle-works/daemon-forks-test-jvms.png" width="40%" height="40%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>At the end of the build, the daemon will do some extra stuff, like executing callbacks, reporting errors, if any, publishing build scans, etc.
After that, the <code class="language-plaintext highlighter-rouge">Gradle Client JVM</code> disconnects from the daemon and exits.
The daemon is now ready for the next build invocation.</p>
<h2 id="whats-next">What’s Next</h2>
<p>In <a href="./how-gradle-works-3">the next blog of the series</a>, we’ll explain what happens under the hood of build script execution.</p>
How Gradle Works Part 1 - Startup2023-01-10T21:00:00-05:00https://blog.gradle.org/how-gradle-works-1Bo Zhang
<p>This is the first blog of a series <em>How Gradle Works</em>, which includes the following topics:</p>
<ul>
<li>How Gradle starts up</li>
<li>How many JVMs are involved in a Gradle build</li>
<li>What happens in each JVM during the build</li>
</ul>
<p>We’ll explain the first topic <em>How Gradle Starts Up</em> in this blog.
Before reading on, we assume you are familiar with basic JVM/Gradle concepts
(jar, classpath, wrapper, daemon, project, task, etc.).</p>
<h2 id="how-gradle-starts-up">How Gradle Starts Up</h2>
<p>There are many ways to start a Gradle build:</p>
<ul>
<li>Local Gradle Distribution in CLI: <code class="language-plaintext highlighter-rouge">/path/to/local/distribution/bin/gradle <SomeTask></code></li>
<li>Gradle Wrapper in CLI: <code class="language-plaintext highlighter-rouge">./gradlew <SomeTask></code></li>
<li>Click a button in your IDE to import a Gradle project or run some tests/tasks</li>
</ul>
<p>What’s the difference? What happens under the hood?
Before we start, we need to remember that Gradle is software running on top of a JVM (<a href="https://en.wikipedia.org/wiki/Java_virtual_machine">Java Virtual Machine</a>).</p>
<h3 id="local-gradle-distribution-in-cli">Local Gradle Distribution in CLI</h3>
<p>You may have downloaded a Gradle distribution to your local computer, either manually or by some package management tools.
Suppose you start a Gradle build by typing <code class="language-plaintext highlighter-rouge">/path/to/local/distribution/bin/gradle <SomeTask></code>. What happens then?</p>
<p>Open the <code class="language-plaintext highlighter-rouge">/path/to/local/distribution/bin/gradle</code> in your favorite text editor, you’ll find that it’s nothing magic but a plain text file,
or more precisely, a shell script file. When you say <code class="language-plaintext highlighter-rouge">/path/to/local/distribution/bin/gradle <SomeTask></code>,
a shell process is started and starts executing the code in the script. At the end of this script, you can see a line:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>exec "$JAVACMD" "$@"
</code></pre></div></div>
<p>So the script’s job is simple: it finds <code class="language-plaintext highlighter-rouge">java</code> command, determines the parameters, and starts a JVM by invoking <code class="language-plaintext highlighter-rouge">exec</code>.
<code class="language-plaintext highlighter-rouge">exec</code> executes a command in the same process, i.e. replaces the current process with a JVM process.</p>
<p><img src="/images/how-gradle-works/local-distribution-startup.png" width="50%" height="50%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>The entry point of <code class="language-plaintext highlighter-rouge">Gradle Client JVM</code> is <a href="https://github.com/gradle/gradle/blob/acc6044325b11874e9626d98dec976a0e495cb62/subprojects/bootstrap/src/main/java/org/gradle/launcher/GradleMain.java"><code class="language-plaintext highlighter-rouge">org.gradle.launcher.GradleMain</code> class</a>.</p>
<p>The JVM started by <code class="language-plaintext highlighter-rouge">exec</code> is a very lightweight JVM. Let’s call it <code class="language-plaintext highlighter-rouge">Gradle Client JVM</code> because it doesn’t run any real build logic.
It searches for a compatible <a href="https://docs.gradle.org/current/userguide/gradle_daemon.html">Gradle Daemon</a> and connects to it via a local socket if found.
If there is no such daemon running, it will start a daemon, which is also a JVM (<code class="language-plaintext highlighter-rouge">Gradle Daemon JVM</code>). Later,</p>
<ol>
<li>The <code class="language-plaintext highlighter-rouge">Gradle Client JVM</code> forwards input (command line arguments, environment variables, stdin, etc.) to the <code class="language-plaintext highlighter-rouge">Gradle Daemon JVM</code>.</li>
<li>The <code class="language-plaintext highlighter-rouge">Gradle Daemon JVM</code> runs the build, and sends output (stdout/stderr) back to the <code class="language-plaintext highlighter-rouge">Gradle Client JVM</code>.</li>
<li>After the build finishes, the <code class="language-plaintext highlighter-rouge">Gradle Client JVM</code> will exit, but the <code class="language-plaintext highlighter-rouge">Gradle Daemon JVM</code> may stay running for some time.</li>
</ol>
<p><img src="/images/how-gradle-works/client-daemon.png" width="50%" height="50%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>However, there is one exception for this workflow. If you pass <code class="language-plaintext highlighter-rouge">--no-daemon</code> to the Gradle build, the client JVM may convert
itself into a daemon JVM if it is <a href="https://docs.gradle.org/current/userguide/gradle_daemon.html#compatibility">compatible with build requirements</a>.
In this case, there is no daemon, no communication, no input/output forwarding at all - the build happens inside the single JVM:</p>
<p><img src="/images/how-gradle-works/no-daemon.png" width="50%" height="50%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>But if <code class="language-plaintext highlighter-rouge">--no-daemon</code> is present and the client JVM is not compatible with build requirements,
a new disposable JVM will still be started for the build and exit at the end of the build:</p>
<p><img src="/images/how-gradle-works/no-daemon-not-compatible.png" width="50%" height="50%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>This is how local Gradle distribution starts up on UNIX OSes. On Windows, the mechanism is very similar - the only difference is that
we invoke <code class="language-plaintext highlighter-rouge">/path/to/distribution/bin/gradle.bat</code> instead of <code class="language-plaintext highlighter-rouge">/path/to/distribution/bin/gradle</code>.</p>
<h3 id="gradle-wrapper-in-cli">Gradle Wrapper in CLI</h3>
<p>As you may know, <a href="https://docs.gradle.org/current/userguide/gradle_wrapper.html">Gradle Wrapper</a> is the recommended way to execute Gradle builds.
What happens when you type <code class="language-plaintext highlighter-rouge">./gradlew <SomeTask></code>? What’s the difference with running from local Gradle distribution?</p>
<p>If you open <code class="language-plaintext highlighter-rouge">gradlew</code> in a text editor, you’ll find it very similar to what we have explained in the last section:
there is no magic, just a shell script. It starts a tiny JVM from <code class="language-plaintext highlighter-rouge">gradle/wrapper/gradle-wrapper.jar</code> in your project.
This JVM will locate or download a specific version of Gradle distribution declared in <code class="language-plaintext highlighter-rouge">gradle/wrapper/gradle-wrapper.properties</code>,
then it will start Gradle inside the same JVM via Java reflection. After that, this tiny JVM acts as the <code class="language-plaintext highlighter-rouge">Gradle Client JVM</code>
in the way explained in the last section.</p>
<p><img src="/images/how-gradle-works/wrapper-daemon.png" width="70%" height="70%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>The entry point of <code class="language-plaintext highlighter-rouge">Gradle Wrapper JVM</code> is the <a href="https://github.com/gradle/gradle/blob/acc6044325b11874e9626d98dec976a0e495cb62/subprojects/wrapper/src/main/java/org/gradle/wrapper/GradleWrapperMain.java"><code class="language-plaintext highlighter-rouge">org.gradle.wrapper.GradleWrapperMain</code> class</a>.</p>
<h3 id="gradle-in-ide">Gradle in IDE</h3>
<p>In the previous sections, <code class="language-plaintext highlighter-rouge">Gradle Client JVM</code> is a dedicated JVM started via CLI.
Actually, it doesn’t have to be a dedicated JVM.
Gradle client can be a part of another JVM, i.e. a JVM that loads some Gradle jars dynamically
and acts as a “Gradle client”: searches/connects to the daemon, starts a new daemon,
and communicates with the daemon.
This programmatic API is called <a href="https://docs.gradle.org/current/userguide/third_party_integration.html#embedding">Tooling API</a>.</p>
<p><img src="/images/how-gradle-works/ide-tapi-daemon.png" width="50%" height="50%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>For example, when you click <code class="language-plaintext highlighter-rouge">Gradle Sync</code> button in IntelliJ IDEA,
IDEA will start a special Gradle build to fetch necessary information (project structure, dependencies, tasks, etc.) of the project via the Tooling API.
Still, all the build logic happens in a <code class="language-plaintext highlighter-rouge">Gradle Daemon JVM</code>, and the Tooling API just reads a build result and returns it to the caller.</p>
<h2 id="whats-next">What’s Next</h2>
<p>In the <a href="./how-gradle-works-2">next blog post of the series</a> we’ll explain what happens inside the <code class="language-plaintext highlighter-rouge">Gradle Daemon JVM</code>.</p>
Introducing Public Design Specs2022-12-07T00:00:00-05:00https://blog.gradle.org/design-spec-annoucementSterling Greene
<p>Gradle Build Tool is an open source project, but a lot of the design work has not been as visible as it could be. Internally, we use design specifications to collaborate on new Gradle features. To make our development process more open and transparent, our design specifications are publicly available via a <a href="https://drive.google.com/drive/folders/1JM7OQwdHGOuoptVtfrwRpzEuzRnCbWXZ?usp=drive_link">Google Drive folder</a>.</p>
<p>This is still a work-in-progress and new documents will be added over time. We welcome community members to take a look. You can comment and provide feedback on all documents still In-Progress. We plan to use these documents to help guide new community contributions. Feel free to link to these documents in GitHub or share them with others.</p>
<p>Along with the <a href="https://github.com/orgs/gradle/projects/31/views/1">public roadmap</a>, we hope opening up the behind-the-scenes work will provide more insight into our future plans.</p>
<p>Please let us know if you have any questions or feedback. You can always reach out to us on <a href="https://gradle-community.slack.com">Community Slack</a>.</p>
Compilation Avoidance2022-11-28T00:00:00-05:00https://blog.gradle.org/compilation-avoidanceAmanda Martin
<p>We’ve recently noticed some community chatter about speeding up Gradle compilation on the JVM by ignoring changes not affecting the ABIs of dependencies. What a great idea! In fact, Gradle has used ABIs for Java out of the box for this without any extra configuration <a href="https://blog.gradle.org/incremental-compiler-avoidance">since version 3.4</a>. We refer to this feature as <strong><a href="https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_compile_avoidance">compilation avoidance</a></strong>. This post explains what ABI-based compilation means for the average workflow. Spoiler: utilizing compilation avoidance is one of the best performance enhancements for any build.</p>
<h4 id="what-is-an-application-binary-interface">What is an application binary interface?</h4>
<p>An application binary interface (ABI) is the interface generated from compiling software that defines internal and external interaction. The ABI represents what is visible to consumers at compile time. When compiling a project, the presence or absence of changes in the ABIs of any of its dependencies determines if compilation is up-to-date or if recompilation is required. These ABIs consist of all the public information about the dependencies that is visible to a consumer project, such as:</p>
<ul>
<li>any public methods with their argument types and the return statements</li>
<li>any public properties and fields</li>
<li>any dependencies used to compile against the ABI.</li>
</ul>
<p>When a person accesses a library in source code, they use an API of the library. When a machine accesses compiled binaries, it uses the ABI.</p>
<h4 id="why-are-abis-relevant-for-build-performance">Why are ABIs relevant for build performance?</h4>
<p>Modern build systems consider ABI compatibility when compiling code to avoid as much compilation as possible when compiling incremental changes to the codebase.</p>
<p>Changes to internal implementation details are <strong><a href="https://docs.gradle.org/current/userguide/caching_java_projects.html#java_compilation">ABI-compatible</a></strong>: they do not change the public interface. In practice, the internal implementation details of a project change much more frequently than the public components. When public information does not change, any downstream projects will not need to be recompiled. Skipping that extra work can have a massive impact on the build performance of a large project, both locally and on CI.</p>
<p>Changes to public interfaces are ABI-incompatible because they change the public interface. ABI-incompatible changes require a recompile of all downstream dependencies. Having to recompile all dependencies downstream of an ABI-incompatible change can massively increase build times.</p>
<h4 id="what-is-compilation-avoidance">What is compilation avoidance?</h4>
<p>Gradle optimizes builds when ABI-compatible changes occur. We call this optimization <a href="https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_compile_avoidance">compilation avoidance</a>.</p>
<p>To see how this works, imagine two projects. <code class="language-plaintext highlighter-rouge">:app</code> depends on <code class="language-plaintext highlighter-rouge">:lib</code>. Here are some things we can generally do to <code class="language-plaintext highlighter-rouge">:lib</code> which are ABI-compatible and do not cause <code class="language-plaintext highlighter-rouge">:app</code> (nor anything that depends on <code class="language-plaintext highlighter-rouge">:app</code>) to need recompiling:</p>
<ul>
<li>Making any change to the body of a method</li>
<li>Adding, removing, or changing private methods, fields, or inner classes</li>
<li>Renaming a parameter</li>
<li>Changing a comment</li>
<li>Changing the name of jars or directories in the classpath</li>
<li>Adding, removing, or changing a resource</li>
</ul>
<p>When ABI-compatible changes happen in <code class="language-plaintext highlighter-rouge">:lib</code>, and a task in ‘:app’ which depends on its classes is run, Gradle doesn’t recompile project <code class="language-plaintext highlighter-rouge">:app</code> or any projects that depend on <code class="language-plaintext highlighter-rouge">:app</code>. In large multi-project builds, this can save large amounts of time.</p>
<h4 id="how-is-compilation-avoidance-different-from-incremental-compilation">How is compilation avoidance different from incremental compilation?</h4>
<p>Not all changes fit the above requirements. In some cases, you will need to change the public ABI of <code class="language-plaintext highlighter-rouge">:lib</code>, which causes a need to compile <code class="language-plaintext highlighter-rouge">:app</code>. Luckily, another out-of-the-box feature called <strong><a href="https://docs.gradle.org/current/userguide/performance.html#incremental_compilation">incremental compilation</a></strong> is used. This will intellegently recompile the classes in <code class="language-plaintext highlighter-rouge">:lib</code> that have changes so the downstream <code class="language-plaintext highlighter-rouge">:app</code> still compiles faster than a full compilation.</p>
<p>Incremental compilation is different from compilation avoidance but very complementary.</p>
<p>“Compilation avoidance” is about <em>avoiding calling the compiler altogether</em> for a given project.</p>
<p>On the other hand, “incremental compilation” <em>does</em> mean calling the compiler, but when doing this making an attempt to <em>reduce the amount of code needed to be recompiled.</em> This is facilitated by keeping track of the references between classes as part of normal compilation, and only recompiling things that are affected by a given change.</p>
<p>With incremental compilation, we look at all classes that have a change compared to compilation avoidance which looks at a project.
Compilation avoidance works across dependent projects, not just inside a project like incremental compilation. That is, incremental compilation optimizes the compilation of individual classes within your projects.
Incremental compilation happens within a single project, but compilation avoidance looks at the relationship between multiple projects. Incremental compilation still saves times across projects as it reduces the amount that needs recompiled.</p>
<h5 id="am-i-using-incremental-compilation-or-compilation-avoidance">Am I using incremental compilation or compilation avoidance?</h5>
<p>In summary if you make an ABI-compatible change, then you are using incremental compilation and compilation avoidance: Incremental compilation for the compile task on the sources where you made the change and compile avoidance for downstream projects.</p>
<p>Alternately, if your change is not ABI-compatible, you can only benefit from incremental compliation.</p>
<h4 id="what-about-abi-jars">What about ABI JARs?</h4>
<p>Some build systems generate ABI JARs to achieve compilation avoidance. Sometimes called header JARs, they have the overall interface without the internal details. The ABI JAR contains only the public methods, fields, constants, and nested types, with all the method bodies removed, and can be used to assess if any changes indicate a need to recompile. With Gradle, we do not need an ABI JAR, as when there is a compiler task we normalize its inputs and generate a unique hash of the ABI. Gradle then uses this hash to check if there are any changes.</p>
<h4 id="notes-for-different-languages">Notes for different languages</h4>
<p>Groovy has an <a href="https://docs.gradle.org/current/userguide/groovy_plugin.html#sec:incremental_groovy_compilation">opt-in experimental feature</a>.</p>
<p>Kotlin has an <a href="https://kotlinlang.org/docs/gradle-compilation-and-caches.html#a-new-approach-to-incremental-compilation">experimental feature</a> developed as part of <a href="https://kotlinlang.org/docs/gradle.html">Kotlin Gradle Plugin</a> by <a href="https://www.jetbrains.com/">JetBrains</a>.</p>
<h4 id="how-can-i-use-this">How can I use this?</h4>
<p>Gradle automatically uses compilation avoidance. Both incremental compilation and compilation avoidance have been enabled by default in Java projects built with Gradle for years. So the next time you dread recompiling your code after an edit, rest assured that Gradle automatically gives you a performance boost.</p>
<p>Not sure if you are using incremental compilation or compilation avoidance, or perhaps you think you have a use case that should work but doesn’t? Find us on <a href="https://join.slack.com/t/gradle-community/shared_invite/zt-1bbiqbuxw-CgB0NeNaK_zuDMEa71A60Q">Slack</a>, we love seeing use cases.</p>
Introducing Test Suites2022-11-15T00:00:00-05:00https://blog.gradle.org/introducing-test-suitesTom Tresansky
<p>As projects grow in size and complexity and otherwise mature, they tend to accumulate a large collection of automated tests.
Testing your software at <a href="https://martinfowler.com/articles/practical-test-pyramid.html">multiple levels of granularity</a> is important to surface problems quickly and to increase developer productivity.</p>
<p>In <a href="https://docs.gradle.org/7.3/release-notes.html#new-features-and-usability-improvements">Gradle 7.3</a>, released November 2021, the Gradle team introduced a new feature called Declarative Test Suites.
Using this feature makes it much easier to manage different types of tests within a single Gradle JVM project without worrying about low level “plumbing” details.</p>
<h2 id="why-test-suites">Why Test Suites?</h2>
<p>Normally - whether or not you’re practicing strict Test Driven Development - as you develop a project you will continuously add new unit tests alongside your production classes.
By convention, for a Java project, these tests live in <code class="language-plaintext highlighter-rouge">src/test/java</code>:</p>
<p><a href="images/introducing-test-suites/default-test-layout.png"><img src="images/introducing-test-suites/default-test-layout.png" alt="Default test directory layout" width="300" height="300" /></a></p>
<p>These unit tests ensure your classes behave correctly in isolation from the very beginning of your project’s lifecycle.
At some point later in development, you will be ready to test how your classes work together to create a larger system using integration tests.
Later still, as a final step in certifying that your project works as designed, you will probably want to run the entire system in end-to-end tests which check functional requirements, measure performance, or otherwise confirm its readiness for release.</p>
<p>There are a lot of error-prone details you need to consider in this process.
Test Suites was created to improve this situation, in response to the hardships detailed below.</p>
<h3 id="considerations-when-setting-up-additional-tests">Considerations when setting up additional tests</h3>
<p>Varied test goals often involve different and incompatible patterns.
At a minimum you’ll want to organize your test code by separating tests into different directories for each goal:</p>
<p><a href="images/introducing-test-suites/adding-other-tests.png"><img src="images/introducing-test-suites/adding-other-tests.png" alt="Test directory layout after adding alternate test types" width="350" height="500" /></a></p>
<p>But separating the source files is only the beginning.
These types of tests might require different preconditions to be met prior to testing, utilize entirely different runtime and compile time dependencies, or interact with different external systems.
The very testing framework (such as JUnit 4 or 5, TestNG or Spock) used to write and run each group of tests could be different.</p>
<p>To correctly model and isolate these differences in Gradle, you’ll need to do more than just separate the tests’ source files.
Each group of tests will require its own:</p>
<ul>
<li>Separate <code class="language-plaintext highlighter-rouge">SourceSet</code>, that will provide a distinct <code class="language-plaintext highlighter-rouge">Configuration</code> for declaring dependencies for the group to compile and run against. You want to avoid leaking unnecessary dependencies across multiple groups of tests, yet still automatically resolve any shared dependencies needed for test compilation and execution.</li>
<li>Support for using different testing frameworks to execute the tests in each group.</li>
<li>A <code class="language-plaintext highlighter-rouge">Test</code> task for each group which might have different task dependencies to provide setup and finalization requirements. You also may want to prevent every type of test from running every time you build the project (for instance, to save any long-running smoke tests for when you think you’re ready to publish).</li>
</ul>
<p>Each component you create in your build scripts to support these requirements must be properly wired into the Gradle project model to avoid unexpected behavior.
Accomplishing this is error-prone, as it requires a thorough understanding of low-level Gradle concepts.
It also requires modifications and additions to multiple blocks of the DSL.</p>
<p>That’s hardly ideal; setting up testing is a single concern and the configuration for it should be co-located and easily discoverable within a buildscript.</p>
<h3 id="wiring-integration-tests-without-test-suites">Wiring integration tests without Test Suites</h3>
<p>It’s helpful to look at a complete example.
Before diving in, take a moment to think about how you would create a separate set of integration tests within a project.</p>
<p>Before Gradle 7.2, the proper way to set up integration tests went like this (note that while this example is written in the Gradle Kotlin DSL, the Groovy setup is very similar):</p>
<p><a href="images/introducing-test-suites/proper-integration-test-setup.png"><img src="images/introducing-test-suites/proper-integration-test-setup.png" alt="Proper integration test setup without test suites" /></a></p>
<ol>
<li>We need to create a <code class="language-plaintext highlighter-rouge">SourceSet</code> that will in turn create the associated <code class="language-plaintext highlighter-rouge">Configuration</code>s we’ll need later. This is low-level plumbing that we shouldn’t have to focus on.</li>
<li>We wire the new test configurations to the test existing configurations, to re-use their dependency declarations. We might not always want to do this.</li>
<li>We need to register a <code class="language-plaintext highlighter-rouge">Test</code> task to run our tests.</li>
<li>We ought to add the new task to the appropriate group and set a description - not technically necessary, but a best practice to make the task discoverable and properly locate it in reports.</li>
<li>We will write our tests using the latest version of JUnit 5.</li>
<li>We need to set up the classpath of our new task - this even more low-level plumbing.</li>
<li>We need to tell our new task where the classes it runs live.</li>
<li>Finally, we add the necessary JUnit dependencies to the built-in test configurations, which our new configurations extend.<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup></li>
<li>The integration tests have an <code class="language-plaintext highlighter-rouge">implementation</code> dependency on the current project’s production classes - this looks somewhat extraneous in this block that also configures the production dependencies for the project.</li>
</ol>
<p>Did you get all that?</p>
<p>The bottom line is that this is simply too complex.
<strong>You shouldn’t have to be a build expert just to set up thorough testing!</strong></p>
<h2 id="test-suites---a-better-way-forward">Test Suites - a better way forward</h2>
<p>Thinking about the difficulties involved in properly handling this scenario, we realized the current situation was inadequate.
We want to support build authors defining multiple groups of tests with different purposes in a declarative fashion while operating at a high level of abstraction.
Although you could previously write your own plugin (or use a pre-existing solution such as <a href="https://nebula-plugins.github.io/">Nebula</a>) to hide the details of this and mitigate the complexity, testing is such an ubiquitous need that we decided to provide a canonical solution within Gradle’s own core set of plugins.
Thus, Test Suites was born.</p>
<p>The <a href="https://docs.gradle.org/7.5.1/userguide/jvm_test_suite_plugin.html">JVM Test Suite Plugin</a> provides a DSL and API to model exactly this situation: multiple groups of automated tests with different purposes that live within a
single JVM-based project.
It is automatically applied by the <code class="language-plaintext highlighter-rouge">java</code> plugin, so when you upgrade to Gradle >= 7.3 you’re already using Test Suites in your JVM projects.
Congratulations!</p>
<p>Here is the previous example, rewritten to take advantage of Test Suites:</p>
<p><a href="images/introducing-test-suites/integration-tests-with-suites.png"><img src="images/introducing-test-suites/integration-tests-with-suites.png" alt="Integration test setup with test suites" /></a></p>
<ol>
<li>All Test Suite configuration is co-located within a new <code class="language-plaintext highlighter-rouge">testing</code> block, of type <a href="https://docs.gradle.org/7.5.1/dsl/org.gradle.testing.base.TestingExtension.html#org.gradle.testing.base.TestingExtension">TestingExtension</a>.</li>
<li>Maintaining backwards compatibility with existing builds that already use the <code class="language-plaintext highlighter-rouge">test</code> task was an important requirement for us when implementing Test Suites. We’ve associated the existing <code class="language-plaintext highlighter-rouge">test</code> task with a default Test Suite that you can use to contain your unit tests.</li>
<li>Instead of representing the relationship between unit tests and integration tests by making their backing <code class="language-plaintext highlighter-rouge">Configuration</code>s extend one another, we keep the machinery powering two types of tests completely separate. This allows for more fine-grained control over what dependencies are needed by each and avoids leaking unnecessary dependencies across test types.<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup></li>
<li>Because each Test Suite could serve very different purposes, we don’t assume they have a dependency on your project (maybe you are testing an external system), so you have to explicitly add one to have access to your production classes in your tests.<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote" rel="footnote">3</a></sup></li>
</ol>
<p>By making use of sensible defaults, Gradle is able to simplify your buildscript significantly.
This script manages to set up a mostly equivalent build as the original but in far fewer lines of code.
Gradle adds a directory to locate your test code and creates the task that run the tests using the suite name as the basis.
In this case, you can run any tests you write located in <code class="language-plaintext highlighter-rouge">src/integrationTestJava</code> by invoking <code class="language-plaintext highlighter-rouge">gradlew integrationTest</code>.</p>
<p>Test Suites aren’t limited to Java projects, either. Groovy, Kotlin, Scala, and other JVM-based languages will work similarly when
their appropriate plugins are applied. These plugins all also automatically apply the JVM Test Suite Plugin, so you can begin adding tests to <code class="language-plaintext highlighter-rouge">src/<SUITE_NAME>/<LANGUAGE_NAME></code> without doing any other configuration.</p>
<h3 id="behind-the-scenes">Behind the scenes</h3>
<p>This short example takes care of all the considerations of the pre-Test Suites example above.<br />
But how does it work?</p>
<p>When you configure a suite in the new DSL, Gradle does the following for you:</p>
<ul>
<li>Creates a Test Suite named <code class="language-plaintext highlighter-rouge">integrationTest</code> (typed as a <a href="https://docs.gradle.org/7.5.1/dsl/org.gradle.api.plugins.jvm.JvmTestSuite.html">JvmTestSuite</a>).</li>
<li>Creates an <code class="language-plaintext highlighter-rouge">SourceSet</code> named <code class="language-plaintext highlighter-rouge">integrationTest</code> containing the source directory <code class="language-plaintext highlighter-rouge">src/java/integrationTest</code>. This will be registered as a test sources directory, so any highlighting performed by your favorite IDE will work as expected.</li>
<li>Creates several <code class="language-plaintext highlighter-rouge">Configuration</code>s derived from the <code class="language-plaintext highlighter-rouge">integrationTest</code> source set, accessible through the Suite’s own <code class="language-plaintext highlighter-rouge">dependencies</code> block: <code class="language-plaintext highlighter-rouge">integrationTestImplementation</code>, <code class="language-plaintext highlighter-rouge">integrationTestCompileOnly</code>, <code class="language-plaintext highlighter-rouge">integrationTestRuntimeOnly</code>, which work like their similarly named <code class="language-plaintext highlighter-rouge">test</code> configurations.</li>
<li>Adds dependencies to these configurations necessary for compiling and running against the default testing framework, which is JUnit Jupiter.</li>
<li>Registers a <code class="language-plaintext highlighter-rouge">Test</code> task named <code class="language-plaintext highlighter-rouge">integrationTest</code> which will run these tests.
The most important difference is that using Test Suites will fully isolate any integration test dependencies from any unit test dependencies.
It also assumes JUnit Platform as the testing engine for new test suites, unless told otherwise.<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">4</a></sup></li>
</ul>
<p>After adding just this minimal block of DSL, you are ready to write integration test classes under <code class="language-plaintext highlighter-rouge">src/integrationTest/java</code> which are completely separate from your unit tests, and to run them via a new <code class="language-plaintext highlighter-rouge">integrationTest</code> task.
No contact with low-level DSL blocks like <code class="language-plaintext highlighter-rouge">configurations</code> is required.</p>
<h2 id="try-it-out-now">Try it out now</h2>
<p>Test Suites is still an <a href="https://docs.gradle.org/7.5.1/userguide/feature_lifecycle.html"><code class="language-plaintext highlighter-rouge">@Incubating</code> feature</a> as we explore and refine the API, but it’s here to stay, and we encourage everyone to try it out now.
For a new project, the easiest way to get started is to use the <a href="https://docs.gradle.org/7.5.1/samples/sample_incubating_jvm_multi_project_with_additional_test_types.html">Gradle Init task</a> and opt-in to using incubating features when prompted; this will generate a sample project using the new DSL.</p>
<h2 id="customizing-your-suites">Customizing your Suites</h2>
<p>The rationale behind Test Suites, just like Gradle in general, is to abstract the details of configuration and use sensible conventions as defaults - but to also allow you to change those defaults as necessary.</p>
<p><a href="images/introducing-test-suites/customizing-test-suites.png"><img src="images/introducing-test-suites/customizing-test-suites.png" alt="Customizing test suites" /></a></p>
<ol>
<li>Configure the built-in Test Suite to use a different testing framework using one of <a href="https://docs.gradle.org/7.5.1/javadoc/org/gradle/api/plugins/jvm/JvmTestSuite.html#useJUnitJupiter--">several convenience methods available</a>.<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote" rel="footnote">5</a></sup></li>
<li>Add a non-project dependency for use in compiling and running a Test Suite.</li>
<li>Add a dependency which is only used when running tests (in this case, a logging implementation).</li>
<li>Access the <code class="language-plaintext highlighter-rouge">integrationTest</code> task which will be created for this Suite to configure it directly (and lazily<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote" rel="footnote">6</a></sup>), within the <code class="language-plaintext highlighter-rouge">testing</code> block.</li>
<li>Define an additional <code class="language-plaintext highlighter-rouge">performanceTest</code> Suite using the bare minimum DSL for a default new Test Suite. Note that this suite will <em>not</em> have access to the project’s own classes, or be wired to run as part of the build without calling its <code class="language-plaintext highlighter-rouge">performanceTest</code> task directly.<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote" rel="footnote">7</a></sup></li>
<li>Suites can be used as task dependencies - this will cause the <code class="language-plaintext highlighter-rouge">check</code> task to depend on the <code class="language-plaintext highlighter-rouge">integrationTest</code> task associated with the <code class="language-plaintext highlighter-rouge">integrationTest</code> Test Suite - the same task we configured in <4>.</li>
</ol>
<p>For more Test Suite custom configuration examples, see the <a href="https://docs.gradle.org/7.5.1/userguide/jvm_test_suite_plugin.html">JVM Test Suite Plugin</a> section of the Gradle user guide.
For adding an additional test suite to a more complex and realistic build, see the <a href="https://docs.gradle.org/7.5.1/samples/sample_incubating_jvm_multi_project_with_additional_test_types.html">Multi-Project sample</a>.</p>
<h2 id="the-future-of-testing-in-gradle">The future of testing in Gradle</h2>
<p>We have many exciting ideas for evolving Test Suites in the future.
One major use case we want to support is multidimensional testing, where the same suite of tests runs repeatedly in different environments (for instance, on different versions or releases of the JVM).
This is the reason for the seemingly extraneous <code class="language-plaintext highlighter-rouge">targets</code> block seen in the examples here and in the user guide.
Doing this will likely involve closer Test Suite integration with <a href="https://docs.gradle.org/7.5.1/userguide/toolchains.html#header">JVM Toolchains</a>.</p>
<p>You’ll also definitely want to check out the <a href="https://docs.gradle.org/7.4/userguide/test_report_aggregation_plugin.html">Test Report Aggregation Plugin</a> added in Gradle 7.4, to see how to easily aggregate the results of multiple <code class="language-plaintext highlighter-rouge">Test</code> task invocations into a single HTML report.
Consolidating test failures and exposing more information about their suites in test reporting is another potential area of future development.</p>
<p>These and other improvements are currently being discussed and implemented, so be sure to keep up to date with this blog and the latest Gradle releases.</p>
<p><em>Happy testing!</em></p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>The version is left off <code class="language-plaintext highlighter-rouge">junit-jupiter-engine</code> because <code class="language-plaintext highlighter-rouge">junit-jupiter-api</code> will manage setting it - but this might look like a mistake. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>In the forthcoming Gradle 8.0, this block will use a new strongly-typed <code class="language-plaintext highlighter-rouge">dependencies</code> API, which should provide a better experience when declaring test dependencies in your favorite IDE. Watch our <a href="https://gradle.org/releases/">releases</a> page for more details. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>Note that in the forthcoming Gradle 7.6, instead of using <code class="language-plaintext highlighter-rouge">project</code> you’ll need to call the new <code class="language-plaintext highlighter-rouge">project()</code> method here. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>The default test suite retains JUnit 4 as its default runner, for compatibility reasons. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>The default test suite is also implicitly granted access to the production source’s implementation dependencies for the same reason. When switching testing frameworks, the new framework’s dependencies are automatically included. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>See the <a href="https://docs.gradle.org/7.5.1/userguide/lazy_configuration.html">Lazy Configuration</a> section of the user guide for more details on lazy configuration. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>Perhaps these tests are meant to exercise a live deployment in a staging environment. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
How We Handle Flaky Tests in Gradle2022-11-02T22:00:00-04:00https://blog.gradle.org/how-we-handle-flaky-tests-in-gradleBo Zhang
<p>Test flakiness is one of the main challenges of automated testing.
Even though Gradle’s focus is increasing developer productivity, the development of Gradle itself suffers from flaky automated tests.
This blog explains some best practices when developing Gradle, which have proved effective over the years we fight with flaky tests.</p>
<h2 id="the-story">The Story</h2>
<p>Like in many other projects, every commit of Gradle must pass tens of thousands of automated tests. Any tiny flakiness may cause developer productivity loss. When I joined Gradle 5 years ago, the CI was full of flaky test failures - people would rerun a build, again and again, hoping to be lucky enough to get a green build.</p>
<p>Later, we started a dedicated developer productivity team to deal with all the flakiness on CI, especially test flakiness.
Here’s how we do it step by step.</p>
<h2 id="retry-failed-tests">Retry Failed Tests</h2>
<p>When a test fails, how do we determine if it’s flaky or not? The easiest way is obviously retrying the failed test immediately:
if the second run succeeds, the failed test is flaky. As a rule of thumb, by simply rerunning the failed test one more time,
the test failures due to flakiness can be reduced by 90%.</p>
<p>Many CI systems can recognize such flaky tests and mark the build as green automatically:</p>
<p><img src="/images/how-we-handle-flaky-tests-in-gradle/teamcity-muted-failure.png" alt="teamcity-muted-failure" /></p>
<p>In this example, the first run fails because of a network error, but the rerun succeeds.
TeamCity recognizes this situation and “mutes” the test failure.</p>
<p>If the build is connected to a <a href="https://gradle.com/">Gradle Enterprise</a> instance and that build has a published <a href="https://scans.gradle.com/">Build Scan</a>,
you can see the flaky tests in the test dashboard:</p>
<p><img src="/images/how-we-handle-flaky-tests-in-gradle/build-scan-flaky-test.png" alt="build-scan-flaky-test" /></p>
<p>In Gradle, we automatically retry the failed test classes with the <a href="https://github.com/gradle/test-retry-gradle-plugin">Test Retry Gradle plugin</a>. Please consult the documentation for how to adopt it in your build.</p>
<h3 id="other-techniques-to-retry-failed-tests">Other Techniques to Retry Failed Tests</h3>
<p>If you are not using Gradle, that’s okay. There are many alternatives that provide similar functionalities:</p>
<ul>
<li><a href="https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html">Maven Surefire Plugin – Rerun failing tests</a></li>
<li><a href="https://junit-pioneer.org/docs/retrying-test/">JUnit Pioneer - Retrying Failing Tests</a></li>
<li><a href="https://bazel.build/reference/be/common-definitions#common-attributes-tests">Bazel <code class="language-plaintext highlighter-rouge">flaky</code> attribute in test rules</a></li>
</ul>
<p>All the tools above are supported by <a href="https://docs.gradle.com/enterprise/flaky-test-detection/">Gradle Enterprise Flaky Test Detection</a>,
which provides awesome <a href="https://gradle.com/gradle-enterprise-solutions/failure-analytics/#test-failure-analytics">Test Failure Analytics</a> features to help you diagnose the test flakiness.</p>
<p><img src="/images/how-we-handle-flaky-tests-in-gradle/flaky-test-trend.png" alt="flaky-test-trend" /></p>
<p>With the flaky test dashboard, you can browse the trend and history of a flaky test case or class. This helps a lot when troubleshooting a flaky test.</p>
<h2 id="quarantine-too-flaky-tests">Quarantine Too-flaky Tests</h2>
<p>Is retrying failed tested enough? The answer is, unfortunately, no. Some tests in the Gradle codebase are so flaky that sometimes they fail even after rerunning two or more times.
These flaky tests are usually caused by flawed production code or test infrastructure, and severely harm developer productivity.</p>
<p>In Gradle, when we find such “too-flaky” tests, we quarantine them. Quarantine means these tests are isolated from the CI pipelines for developer feedback.
In other words, they disappear from developers’ sight and won’t block developers anymore.</p>
<p>Where do they go? Do we lose the test coverage for these “too-flaky” tests? No, they are collected and executed in a nightly CI job (we call it <code class="language-plaintext highlighter-rouge">Flaky Test Quarantine</code>) that runs once per day, like this:</p>
<p><img src="/images/how-we-handle-flaky-tests-in-gradle/steps.png" alt="steps" /></p>
<p>The “too-flaky” tests are quarantined by manually adding a custom <a href="https://github.com/gradle/gradle/blob/4092a64b01f281016039ac627d4d617302d707dd/subprojects/internal-testing/src/main/groovy/org/gradle/test/fixtures/Flaky.groovy#L39"><code class="language-plaintext highlighter-rouge">@Flaky</code> annotation</a> onto the test class or method once we see any occurrence.
We also introduced <a href="https://github.com/gradle/gradle/blob/c2707be809b655580a0aeeaaca4b1cb9211778df/build-logic/basics/src/main/kotlin/gradlebuild/basics/BuildParams.kt#L69">three strategies</a> to select the tests to run:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">EXCLUDE</code>: All tests excluding the <code class="language-plaintext highlighter-rouge">@Flaky</code> tests are selected to run. This is the default strategy for normal CI pipelines.</li>
<li><code class="language-plaintext highlighter-rouge">ONLY</code>: Only <code class="language-plaintext highlighter-rouge">@Flaky</code> tests are selected to run. This is the strategy for <code class="language-plaintext highlighter-rouge">Flaky Test Qurantine</code> job.</li>
<li><code class="language-plaintext highlighter-rouge">INCLUDE</code>: All tests including the <code class="language-plaintext highlighter-rouge">@Flaky</code> tests are selected to run.</li>
</ul>
<p>(For how to implement the strategies with different test frameworks, please see the Javadoc in <a href="https://github.com/gradle/gradle/blob/4092a64b01f281016039ac627d4d617302d707dd/subprojects/internal-testing/src/main/groovy/org/gradle/test/fixtures/Flaky.groovy"><code class="language-plaintext highlighter-rouge">@Flaky</code> annotation</a>.)</p>
<h2 id="fix-flaky-tests">Fix Flaky Tests</h2>
<p>Everything above doesn’t <strong>solve</strong> the real problems in flaky tests - it only <strong>hides</strong> the problems and gives you a false sense of security.
Action must be taken to fix the flawed code, i.e. the root causes of the flaky tests.</p>
<p>Thanks to the test dashboard in <a href="https://docs.gradle.com/enterprise/flaky-test-detection/">Gradle Enterprise Flaky Test Detection</a> feature, we can easily:</p>
<ul>
<li>Count the number of all flaky tests.</li>
<li>Order all flaky tests by the frequency they happen.</li>
<li>Browse the history of a specific flaky test. This is important because we can easily identify the person who introduced the flaky test.</li>
</ul>
<p>We review all flaky tests once per week and take action to keep their total number as low as possible. Here’s how we do the review:</p>
<p><img src="/images/how-we-handle-flaky-tests-in-gradle/flow-chart.png" width="50%" height="50%" style="margin-left: auto; margin-right: auto; display: block" /></p>
<p>In this way, most new flaky tests are noticed and assigned immediately. The flaky tests that can’t be assigned will be collected and fixed on “flaky fix-it day.”
The “flaky fix-it day” is a 1 or 2-day hackathon we organize when the flaky test number exceeds a certain threshold (for example, 1% of the total test number).
During flaky fix-it day, all dev teams work together and focus on fixing accumulated flaky tests.</p>
<h2 id="summary">Summary</h2>
<p>Flaky tests are painful, but measures can be taken to prevent them from harming developer productivity.
This blog describes the measures we take in the development of Gradle to get flaky tests under control: retry all failed tests, quarantine the too-flaky tests,
and fix the flaky tests through teamwork.</p>
<h2 id="feedback">Feedback</h2>
<p>Let us know if you have any questions on our <a href="https://discuss.gradle.org/">forums</a> or <a href="https://gradle.com/slack-invite">Gradle Community Slack</a>.</p>
A Better Way to Use Gradle With Github Actions2022-09-14T02:00:00-04:00https://blog.gradle.org/gh-actionsDaz DeBoer
<h2 id="running-gradle-builds-on-github-actions">Running Gradle builds on GitHub Actions</h2>
<p>GitHub Actions provides a convenient and powerful CI platform for projects hosted on GitHub. To enhance the experience of building Gradle projects on GitHub Actions, the Gradle team has developed the <code class="language-plaintext highlighter-rouge">gradle-build-action</code>. Together with Gradle Build Scans™, the <code class="language-plaintext highlighter-rouge">gradle-build-action</code> provides deep integration between Gradle and GitHub Actions, providing easier setup and a better experience when building and testing your Gradle project with GitHub Actions.</p>
<p>The <code class="language-plaintext highlighter-rouge">gradle-build-action</code> is the <a href="https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-gradle">officially supported</a> way to run your Gradle build in GitHub Actions, taking care of preparing and optimizing Gradle for your GitHub Actions workflow. When applied to a workflow, all subsequent Gradle invocations will be optimized, allowing you to simply run <code class="language-plaintext highlighter-rouge">./gradlew build</code> in a regular workflow step.</p>
<p>The primary features provided by the <code class="language-plaintext highlighter-rouge">gradle-build-action</code> are:</p>
<ul>
<li>Download and install a specified version of Gradle (if required)</li>
<li>Save and restore the Gradle User Home directory between workflow runs</li>
<li>Display results and Gradle Build Scans for any Gradle build executed in your workflow</li>
</ul>
<p>Each of these features is described in more detail below.</p>
<h4 id="using-the-gradle-build-action">Using the Gradle Build Action</h4>
<p>It’s easy to take advantage of the <code class="language-plaintext highlighter-rouge">gradle-build-action</code> by adding a setup step to your existing GitHub Actions workflow, similar to the way you would use the <a href="https://github.com/actions/setup-java#setup-java">actions/setup-java</a> action to configure a JVM for your workflow.</p>
<p><img width="797" alt="gh-actions1" src="https://user-images.githubusercontent.com/51727488/189988870-04497d32-65d2-47e5-b197-94f4e34c5b75.png" /></p>
<p>For details on how to use the <code class="language-plaintext highlighter-rouge">gradle-build-action</code> in your workflow, check out the <a href="https://docs.gradle.org/current/userguide/github-actions.html#sec:configure_github_actions">Gradle Build Tool documentation</a> or the <a href="https://github.com/gradle/gradle-build-action#use-the-action-to-setup-gradle">README</a>.</p>
<h3 id="features-of-the-gradle-build-action">Features of the Gradle Build Action</h3>
<h4 id="save-and-restore-the-gradle-user-home-directory-between-workflow-runs">Save and restore the Gradle User Home directory between workflow runs</h4>
<p>The Gradle User Home directory persists a lot of valuable state that speeds up subsequent invocations of Gradle. Downloaded dependencies, compiled build scripts, the local Build Cache and generated API jars are all stored in the Gradle User Home.</p>
<p>Each GitHub Actions Job runs on a fresh build runner, meaning it will start with an empty Gradle User Home directory. Fortunately, GitHub Actions provides a caching mechanism that allows the Gradle User Home state to be saved after running one job, and restored before running another. This can greatly reduce the time spent running your Gradle builds, by removing the need to re-download dependencies, re-generate Gradle API jars, etc.</p>
<p>Getting this caching right is not trivial, however. It’s easy to save too much state (quickly filling the GitHub Actions cache), or underspecify the cache key (leading to important things being omitted).</p>
<p>The <code class="language-plaintext highlighter-rouge">gradle-build-action</code> takes care of the details for you, saving and restoring the most important files from Gradle User Home in a Job-specific manner, while extracting common files and saving them separately in order to reduce redundancy in the entries stored. Much of the behaviour is configurable and tweakable. Details of the Save/Restore strategy employed by <code class="language-plaintext highlighter-rouge">gradle-build-action</code> (and the related configuration options) will be the topic of a future post.</p>
<h4 id="display-outcome-and-gradle-build-scan-for-every-build">Display outcome and Gradle Build Scan for every build</h4>
<p>By default, GitHub Actions does not integrate deeply with Gradle in the way some other CI platforms do. It provides no high-level view of build outcomes, or nicely rendered display of tests passed and failed. Instead, you are required to inspect the logs for any failure messages, and test reports need to be saved as build artifacts which can be downloaded to inspect any test failures.</p>
<p>However, by using the <code class="language-plaintext highlighter-rouge">gradle-build-action</code> and by configuring each Gradle invocation to publish Build Scans, you can set up your GitHub Actions workflow to provide these usability features you have grown to expect. The <code class="language-plaintext highlighter-rouge">gradle-build-action</code> instruments all Gradle build invocations, capturing details such as tasks executed, build outcome and the link to any Build Scan produced. And the linked Gradle Build Scan provides a complete view of a build execution, including all build logs, a complete task timeline, test outputs, dependencies resolved and performance characteristics of your build.</p>
<p><img width="806" alt="gh-actions2" src="https://user-images.githubusercontent.com/51727488/189988848-386c3d6b-d93d-4b4b-8c76-3f62a13ba3bf.png" /></p>
<p>For more details, read about <a href="https://gradle.com/blog/determine-the-root-cause-of-github-actions-failures-faster-with-gradle-enterprise/">how the Google AndroidX team leverages the gradle-build-action and Gradle Build Scans</a> to help them troubleshoot build and test failures faster in their GitHub Actions CI environment.</p>
<h4 id="execute-your-build-using-a-specified-gradle-version">Execute your build using a specified Gradle version</h4>
<p>We recommend that most projects use the <a href="https://docs.gradle.org/current/userguide/gradle_wrapper.html">Gradle Wrapper</a> to run Gradle, and by default the <code class="language-plaintext highlighter-rouge">gradle-build-action</code> will not download or install a Gradle Distribution. But there are certain situations where you may want to run your build with a different version of Gradle: e.g. you want to smoke test your build with a Gradle release-candidate, or you want to test your Plugin samples with a matrix of different Gradle versions.</p>
<p>For this purpose, the <code class="language-plaintext highlighter-rouge">gradle-build-action</code> is able to download and install any version of Gradle. If a ‘gradle-version’ parameter is supplied to the action, it will download (and cache) the Gradle distribution zip, install it, and add it to the PATH. Alias values like ‘latest’, ‘release-candidate’ and ‘nightly’ are also supported. This installed version will then be used when you run <code class="language-plaintext highlighter-rouge">gradle</code> in a subsequent workflow step.</p>
<h3 id="why-use-the-gradle-build-action">Why use the gradle-build-action?</h3>
<p>If you’re familiar with GitHub Actions, you may be wondering why you’d want to use the <code class="language-plaintext highlighter-rouge">gradle-build-action</code> instead of simply configuring <code class="language-plaintext highlighter-rouge">actions/cache</code> or <code class="language-plaintext highlighter-rouge">actions/setup-java</code> to save and restore Gradle dependencies between workflow runs.</p>
<p>We believe the <code class="language-plaintext highlighter-rouge">gradle-build-action</code> offers substantial benefits: please check out <a href="https://github.com/gradle/gradle-build-action#why-use-the-gradle-build-action">this section of the action README</a>. If you still have questions, feel free to ask at <a href="https://github.com/gradle/gradle-build-action/issues">https://github.com/gradle/gradle-build-action/issues</a>.</p>
<h3 id="a-better-experience-with-gradle-and-github-actions">A better experience with Gradle and GitHub Actions</h3>
<p>The <code class="language-plaintext highlighter-rouge">gradle-build-action</code> provides a simple and effective way to run Gradle builds on GitHub Actions, maximizing the use of the GitHub Actions cache and presenting important information about each Gradle execution. When combined with free Gradle Build Scans (or self-hosted Gradle Enterprise Build Scans), running Gradle builds on GitHub Actions is better than ever.</p>
Plugin Portal Potential Data Exposure2022-08-24T00:00:00-04:00https://blog.gradle.org/portal-information-exposureLouis Jacomet
<p>On 16th August 2022, Gradle Plugin Portal and the Gradle Discourse forums were impacted by a security incident that could have led to exposure of the personal data of some Gradle community members.</p>
<p>No other services, hosted on <a href="https://gradle.org">gradle.org</a>, <a href="https://gradle.com">gradle.com</a>, or elsewhere were impacted.</p>
<h1 id="what-happened">What happened?</h1>
<p>An Amazon Web Services key granting access to database backups that contained personal information for a subset of the Gradle Plugin Portal and Discourse forum users was exposed in a Git commit.
This key was exposed for two hours before being revoked.</p>
<p>We believe it is unlikely that the exposure led to an unauthorized access of data, but we are taking actions and notifying impacted individuals out of caution and transparency.</p>
<h1 id="what-data-was-exposed">What data was exposed?</h1>
<p>We have inspected the database backups to determine what data could have been accessed.
The backups were from the development environment that contained data for the Gradle Plugin Portal and Discourse users.</p>
<p>We are emailing affected users to notify them specifically of this incident.
If you do not receive an email from us about this incident, you are not affected by it.</p>
<p>The personal data that could have been impacted by the incident likely varies depending on the user but could include information such as email address, display names, and for some individuals, hashed and salted passwords.
We have confirmed that no activity has occurred with respect to any account related to any of the hashed and salted passwords potentially impacted by the incident.
The following data could have been exposed:</p>
<h4 id="email-users">7133 display names, usernames and potential GitHub usernames and email addresses for all users that were present in the development database of the Gradle Plugin Portal</h4>
<ul>
<li>Display name and username are already public on the Plugin Portal or Discourse. So is the association with the GitHub username when using GitHub as the identity provider.</li>
<li>However, emails are not public on the Plugin Portal or Discourse.
If the database was downloaded by a third party, they would be able to link these 7000 Plugin Portal or Discourse usernames to an email address.</li>
</ul>
<h4 id="email-migration">3994 usernames and email addresses from a historical version of the Gradle Build Tool Forum</h4>
<ul>
<li>Usernames (mixture of handles and display names) are already public on our forums</li>
<li>However, email addresses are not public.
If the database was downloaded by a third party, they would be able to link these 4008 usernames to an email address.</li>
</ul>
<h4 id="password-unactivated">195 username, email and hashed and salted passwords for un-activated user accounts</h4>
<ul>
<li>Hashes were generated using bcrypt</li>
<li>None of these users have been activated since the incident</li>
<li>Only 2 of those hashed passwords match production users</li>
</ul>
<h1 id="what-data-remained-safe">What data remained safe?</h1>
<ul>
<li>None of the publishing keys for any plugin on the production Gradle Plugin Portal were exposed.</li>
<li>None of the Plugin Portal artifacts could be replaced.</li>
<li>None of the Discourse forums posts could be altered.</li>
</ul>
<h1 id="what-should-you-do-to-protect-yourself-from-data-abuse">What should you do to protect yourself from data abuse?</h1>
<ul>
<li>We recommend being extra cautious around Gradle-themed phishing attacks that may seek to target your status as a Gradle plugin author or Gradle forum user.
If you have any questions about whether a communication is authentically from Gradle, feel free to reach out to us at <code class="language-plaintext highlighter-rouge">security@gradle.com</code>.</li>
<li>We recommend using unique and challenging passwords for all of your accounts (Gradle-related or otherwise).
Best practice is to use password-manager generated unique passwords for each service.</li>
<li>Reach out to us if you have any questions or would like to change the email address of your account by contacting us at <code class="language-plaintext highlighter-rouge">plugin-portal-support@gradle.com</code>.</li>
</ul>
<h1 id="what-have-we-done-to-respond-to-the-incident">What have we done to respond to the incident?</h1>
<ul>
<li>Carried out incident response procedures, including investigating and mitigating the exposure.</li>
<li>Enabled Github Push Protection across all repositories organization-wide to prevent secret keys from being accidentally pushed to GitHub</li>
<li>Notified potentially affected community members</li>
<li>Stale pending user activations have been purged</li>
</ul>
<h1 id="what-will-we-do-to-prevent-further-incidents">What will we do to prevent further incidents?</h1>
<p>We are taking the following further steps to improve our security and avoid repeat incidents:</p>
<ul>
<li>Stopping passing credentials in command-line arguments for tests</li>
<li>Introducing retention policies on our database</li>
<li>Cleaning up the development and production databases, deleting unneeded data</li>
<li>Enabling S3 access logging account-wide by default</li>
<li>Encrypting database backups</li>
</ul>
<h1 id="incident-timeline">Incident Timeline</h1>
<dl>
<dt>2022.08.16 at 10:03 UTC</dt>
<dd>Commit with AWS access key and secret was pushed</dd>
<dt>2022.08.16 at 10:09 UTC</dt>
<dd>AWS notified Gradle of this disclosure and automatically applied a quarantine policy that prevents destructive actions from being taken with this key</dd>
<dt>2022.08.16 at 12:04 UTC</dt>
<dd>AWS key disabled by Gradle staff</dd>
<dt>2022.08.16 - 2022.08.23</dt>
<dd>Internal incident response</dd>
<dt>2022.08.24</dt>
<dd>Publication of this blog entry and notifications of affected users</dd>
</dl>
<h1 id="final-words">Final words</h1>
<p>We are sorry to have been in a position of potentially disclosing information entrusted to us.
We are using this incident as a way to improve our internal systems and security to reduce the likelihood of such an event happening again.</p>
Simplifying the Plugin-Publish Plugin2022-07-18T00:00:00-04:00https://blog.gradle.org/simplifying-the-plugin-publish-pluginJozsef Bartok
<p>Configuring the publication of Gradle plugins to the <a href="https://plugins.gradle.org/">Portal</a> happens with the help of the
<a href="https://plugins.gradle.org/plugin/com.gradle.plugin-publish">Plugin-Publish plugin</a>.
The recently released <a href="https://plugins.gradle.org/plugin/com.gradle.plugin-publish/1.0.0">version 1.0.0</a> of the plugin
significantly improves the process by having stronger opinions and a more straightforward configuration.</p>
<h1 id="stronger-conventions">Stronger Conventions</h1>
<p>Gradle plugin development has a long history and has gradually evolved to use helper plugins to aid you with setting up your plugin project.
The most important ones:</p>
<ul>
<li><a href="https://docs.gradle.org/current/userguide/java_gradle_plugin.html">Gradle Plugin Development plugin</a> (<code class="language-plaintext highlighter-rouge">java-gradle-plugin</code>)</li>
<li><a href="https://docs.gradle.org/current/userguide/publishing_maven.html">Maven Publish plugin</a> (<code class="language-plaintext highlighter-rouge">maven-publish</code>)</li>
</ul>
<p>The Plugin-Publish plugin shares this long history, and it ended up supporting all the various combinations of helper plugins.
This results in too many ways of publishing plugins, and too many configuration options, with some having hard-to-understand semantics depending on the exact setup.</p>
<p>For example, publication metadata can be generated via the <code class="language-plaintext highlighter-rouge">maven-publish</code> plugin or in other, ad-hoc ways.
Depending on this, both plugin and marker GAV coordinates can vary, with surprising effects.</p>
<p><a href="https://plugins.gradle.org/plugin/com.gradle.plugin-publish/1.0.0">Version 1.0.0</a> of the Plugin-Publish plugin addresses
these issues by tightening and enforcing the conventions. The main helper plugins are always auto-applied:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">maven-publish</code> is always used for generating the publication metadata (Maven POM & Gradle Module Metadata).</li>
<li><code class="language-plaintext highlighter-rouge">java-gradle-plugin</code> is always applied and used as a base for plugin definitions;
as a consequence, the <code class="language-plaintext highlighter-rouge">gradlePlugin</code> block is now mandatory and the <code class="language-plaintext highlighter-rouge">pluginBundle</code> block has been trimmed down to remove all duplication.</li>
</ul>
<h1 id="unambiguous-configuration">Unambiguous configuration</h1>
<p>When configuring the publication of plugins, each property of the end result should have a single way of being specified.</p>
<p>To ensure that’s the case, <a href="https://plugins.gradle.org/plugin/com.gradle.plugin-publish/1.0.0">version 1.0.0</a> of the Plugin-Publish plugin is removing the <code class="language-plaintext highlighter-rouge">pluginBundle.plugins</code> block, as it was duplicating <code class="language-plaintext highlighter-rouge">gradlePlugin.plugins</code>.</p>
<p>Unfortunately, the duplication is not (yet) perfect, so the <code class="language-plaintext highlighter-rouge">pluginBundle.pluginTags</code> map needs to be added to be able to define different tags for plugins published together.
The more elegant solution is to allow tags to be specified in the elements of the <code class="language-plaintext highlighter-rouge">gradlePlugin.plugins</code> block, but this will be possible only starting with Gradle <code class="language-plaintext highlighter-rouge">7.6</code>.</p>
<p>See an example <a href="https://plugins.gradle.org/docs/publish-plugin#examples">here</a> to be updated with further improvements once Gradle <code class="language-plaintext highlighter-rouge">7.6</code> is out.</p>
<h1 id="clearly-defined-interactions">Clearly defined interactions</h1>
<p><a href="https://plugins.gradle.org/plugin/com.gradle.plugin-publish/1.0.0">Version 1.0.0</a> specifies (and automates) the way the Plugin-Publish plugin interacts with other frequently used plugins:</p>
<ul>
<li>If the <a href="https://docs.gradle.org/current/userguide/signing_plugin.html">Signing plugin</a> is applied, then all published artifacts of the plugin will be automatically signed.</li>
<li>If the <a href="https://github.com/johnrengelman/shadow">Shadow plugin</a> is applied, then the plugin’s main jar will be the shadow jar, as configured by the Shadow plugin.</li>
</ul>
<p>Should those plugins be applied, the user receives a notification on the console about the automatic configuration they trigger.</p>
<h1 id="breaking-changes">Breaking changes</h1>
<p>In the spirit of stronger conventions and simplified configuration <a href="https://plugins.gradle.org/plugin/com.gradle.plugin-publish/1.0.0">version 1.0.0</a>
also comes with some breaking changes.</p>
<p>The lowest Gradle version supported by the Plugin-Publish plugin becomes <code class="language-plaintext highlighter-rouge">4.10.3</code>. It has been <code class="language-plaintext highlighter-rouge">3.5.1</code> previously.</p>
<p>Making the <code class="language-plaintext highlighter-rouge">maven-publish</code> plugin mandatory also results in the <code class="language-plaintext highlighter-rouge">mavenCoordinates</code> and <code class="language-plaintext highlighter-rouge">withDependencies</code> blocks being completely removed.
The functionality they offered is no longer needed or meaningful.</p>
<h1 id="documentation">Documentation</h1>
<p>The main documentation for using the <code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code> plugin is <a href="https://plugins.gradle.org/docs/publish-plugin">on the Plugin Portal</a>.
There is <a href="https://docs.gradle.org/current/userguide/publishing_gradle_plugins.html">additional documentation in the Gradle Manual</a>, but please note that the manual will focus on this new version of the <code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code> only starting with Gradle <code class="language-plaintext highlighter-rouge">7.6</code>.</p>
<h1 id="conclusion">Conclusion</h1>
<p>With this simplified and clarified setup, we encourage plugin authors to adopt this new version as soon as possible.</p>
<p>We are looking for your feedback on our <a href="https://gradle.com/slack-invite">Community Slack</a> or the <a href="https://discuss.gradle.org/">Gradle Forums</a>.
Don’t hesitate to contact us both if you encounter problems with the new version or if you want to express your support for the work we do.</p>
General Build Distribution: A Game-Changer or a Gimmick?2022-07-12T02:00:01-04:00https://blog.gradle.org/general-build-distributionKyle Moore
<p>The <a href="/remote-and-distributed-build-patterns">Remote and Distributed Build Patterns</a> article explains the differences between remote and distributed builds and variations on each. Specifically, we distinguished between “test distribution” and “general build distribution”.</p>
<p>This article discusses distributed builds in a broader perspective of improving build feedback times. We’ll start by explaining the types of changes engineers tend to make, identify the typical bottlenecks and share how these relate to distributed builds. We will also study the performance potential of general build distribution. Finally, we will explore a holistic approach to improving build feedback times.</p>
<p>In greater detail below, we will elaborate on these three findings:</p>
<ul>
<li>Building in a distributed fashion is not a substitute for a well-tuned build process.</li>
<li>Improving incremental build performance, not “full rebuilds”, is the most important aspect of improving the local developer experience.</li>
<li>General build distribution of a well-tuned build beyond test distribution is an evolutionary, not revolutionary, process that yields marginal performance benefits for most JVM projects.</li>
</ul>
<p>The analysis and findings presented here apply especially to projects for the JVM ecosystem. Future follow-up articles will address the Android and native/iOS ecosystems.</p>
<h2 id="most-important-scenarios-to-optimize">Most Important Scenarios to Optimize</h2>
<p>Two keys to improving the local developer experience lie in understanding the typical bottlenecks faced by engineers, as well as the types of changes built by engineers as they add new features, fix bugs and write tests.</p>
<h3 id="test-execution-is-the-bottleneck">Test Execution is the Bottleneck</h3>
<p>Test execution is frequently the single most time-consuming portion of build time. Optimizing builds to avoid unnecessary test execution can yield large productivity gains. The Gradle Build Tool already skips tests when no meaningful changes are detected on the classpath, and can also restore test execution results from the build cache. The post <a href="https://www.stefan-oehme.com/stop-rerunning-tests">Stop rerunning your tests</a> does a great job of explaining the efficiencies available by minimizing test re-execution.</p>
<p>In the context of distributed builds, this bottleneck is addressed by <a href="/remote-and-distributed-build-patterns#modern-test-distribution">modern test distribution</a> such as the Test Distribution solution provided by Gradle Enterprise.</p>
<h3 id="frequency-of-incremental-builds-vs-full-rebuilds">Frequency of Incremental Builds vs. Full Rebuilds</h3>
<p>The next key point is that in the vast majority of cases engineers are building small, incremental changes. We posit that these small, incremental changes are unlikely to benefit from distributing the build steps beyond test distribution. Also, developers using modern build systems seldom perform a “full rebuild” without the benefit of a shared build cache or retained history of a previous build on the same machine.</p>
<p>Consider a change to the body of a private method of a Java class: only that class must be recompiled and the library containing it is reassembled. But there is no reason to recompile a downstream consumer of that library as it cannot link to a private method. At the opposite end of the spectrum, consider a modification to the public API of a “common” library consumed by many other subprojects in a multiproject build. This will cause a “domino effect” by causing its downstream consumers to be recompiled. General build distribution may help in this scenario, but we contend this is the exception, not the norm (see <a href="#parallelization-factor">Parallelization factor</a> below for more insights).</p>
<p>Additionally, Java compilation is relatively fast compared to “native” languages, further reducing the optimization potential of general build distribution in Java projects.</p>
<p>Thus, we encourage skepticism at claims of building large projects “from scratch” as a true measure of build system performance, or as a justification for implementing a remote or distributed build.</p>
<h2 id="parallelization-factor">Parallelization factor</h2>
<p>The key to understanding the maximum speed potential of any build (locally built, hosted remotely or distributed) is to visualize the interdependencies of its outputs. Imagine a relatively small software project having three subprojects: A, B and C. If compiling subproject C requires the outputs of subprojects A and B, then C <em>depends on</em> A and B. Most importantly, we cannot begin building C until both A and B are complete; therefore the best-case build-time scenario can be denoted as <em><strong>max(A, B) + C</strong></em>. Given a local or remote build host with unlimited CPU cores, or a pool of unlimited distributed build agents, the build <em>cannot</em> be parallelized further than this bottleneck.</p>
<p><img src="/images/remote-and-distributed-builds/fan-out.svg" alt="Example project structure" />
<em>Figure 1.: Project structure</em></p>
<p>As we see that this bottleneck is dependency-based and not performance-based, we now have the ability to predict the potential benefit of remote or distributed builds.</p>
<p>To put this theory to the test, we’ve performed some analysis<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> of the <em>parallelization factor</em> to establish a theoretically achievable minimal build time given the bottlenecks described above. We examined the build of Gradle itself and other sizable builds<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup> in collaboration with some of our partners. We’ve found these interesting results:</p>
<ul>
<li>Test execution consumes the vast majority of build time, accounting for 80-90% of the end-to-end CI cycle.</li>
<li>After test execution, the most time-consuming tasks are CPU-intensive tasks like compilation or validation, followed by disk-bound packaging/assembly tasks.</li>
<li>More than half of the non-test tasks are executed in a single process.</li>
</ul>
<p>This last point is critical: tasks that operate as a single process - with no other processes executing simultaneously - are indicative of a bottleneck, like subproject C in the above example. Single-process tasks are proof that further optimization via distribution is not possible. It <em>is</em> possible that a more powerful remote CPU would finish the compilation task more quickly, but this benefit could easily be negated by the overhead of sending bits back and forth.</p>
<p><img src="/images/remote-and-distributed-builds/parallelism-factor.svg" alt="Cumulative work time" />
<br />
<em>Figure 2.: Cumulative work time, grouped by number of concurrent workers. Half of the work was executed with no other busy processes running in parallel.</em></p>
<p>Setting aside test execution (addressed by Test Distribution, see <a href="#test-execution-is-the-bottleneck">above</a>), and focusing on the remaining CPU-intensive 10-20% portion of build times, we find that the potential for optimization is low. The failure of half these tasks to execute in parallel with other processes means that, at best, a general distribution solution could expedite only 5-10% of overall build time, while incurring significant costs in terms of build complexity and management overhead.</p>
<h2 id="the-path-forward">The Path Forward</h2>
<p>As we discussed above, most of the changes done by developers are small, incremental changes and the biggest bottleneck is typically test execution. Therefore, focusing the build optimizations on those aspects will typically yield the best results. The following section lists some of the key steps your build process can implement today. These fundamentals of build performance optimization will not just improve any build whether local, remote or distributed, but will also ensure the best possible performance when potentially moving to a remote or distributed environment in the future.</p>
<p>In this order, we recommend taking advantage of these Gradle Build Tool features to optimize local build feedback time. Most of these features are documented in further detail at <a href="https://docs.gradle.org/current/userguide/performance.html">Improving the Performance of Gradle Builds</a>:</p>
<ol>
<li><a href="https://docs.gradle.org/current/userguide/performance.html#incremental_build">Incremental Build</a></li>
<li><a href="https://docs.gradle.org/current/userguide/performance.html#compile_avoidance">Compile Avoidance</a> and <a href="https://docs.gradle.org/current/userguide/performance.html#incremental_compilation">Incremental Compilation</a></li>
<li><a href="https://docs.gradle.org/current/userguide/build_cache.html">Remote Build Cache</a></li>
<li><a href="https://docs.gradle.org/current/userguide/performance.html#parallel_execution">Parallel Execution</a></li>
<li><a href="https://docs.gradle.org/current/userguide/configuration_cache.html">Configuration Cache</a> (also increases local parallelism)</li>
</ol>
<p>Additionally, the following features in Gradle Enterprise drastically shorten test feedback time which is usually by far the biggest bottleneck in build performance:</p>
<ul>
<li><a href="https://gradle.com/gradle-enterprise-solutions/test-distribution/">Test Distribution</a></li>
<li><a href="https://gradle.com/gradle-enterprise-solutions/predictive-test-selection/">Predictive Test Selection</a></li>
</ul>
<p>While general build distribution may demonstrate impressive build performance gains when measured in isolation, we’ve demonstrated that for most JVM projects it’s unlikely to offer significant additional build performance improvements for typical scenarios in well-optimized builds. This is not to suggest that we find a general distribution solution uninteresting. Rather, we view this on our long-term roadmap as an evolutionary, not revolutionary, solution.</p>
<h2 id="summary">Summary</h2>
<p>Anecdotal evidence and industry experience have shown two things: first, engineers are most likely to iterate and rebuild small, incremental changes - not rebuilding the entire project from scratch. Second, regardless of the type of change being built, test execution is the primary cause of build slowness and reduced developer productivity.</p>
<p>Using active process counts as a proxy for the potential parallelization of local builds, we’ve shown that general build distribution solutions would have a relatively small - if any - impact on build performance for many builds in the JVM ecosystem.</p>
<p>Running all aspects of a JVM build in a purely distributed fashion is not a panacea. Existing Gradle Build Tool features like incremental task execution, compilation avoidance, incremental compilation, build cache and configuration cache are available today and greatly reduce build times, especially for the most frequent incremental changes. Additionally, commercial features in Gradle Enterprise like <a href="https://gradle.com/gradle-enterprise-solutions/test-distribution/">Test Distribution</a> and <a href="https://gradle.com/gradle-enterprise-solutions/predictive-test-selection/">Predictive Test Selection</a> drastically reduce test execution time which is the primary bottleneck for most builds.</p>
<h2 id="feedback">Feedback</h2>
<p>Let us know if you have any questions on our <a href="https://discuss.gradle.org/">forums</a> or <a href="https://gradle.com/slack-invite">Gradle Community Slack</a>.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>We used this <a href="https://github.com/lptr/analyze-builds">tool</a> to gather 30 days of Build Scan™ data from a selected pool of Gradle Enterprise servers. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>The servers captured data for the builds of Gradle, the Gradle Enterprise commercial product, the <a href="https://ge.spring.io">Spring</a> project, and a corporation building thousands of microservices using Gradle. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Remote and Distributed Build Patterns2022-07-12T02:00:00-04:00https://blog.gradle.org/remote-and-distributed-build-patternsKyle Moore
<p>A frequently-requested feature for the Gradle Build Tool is the ability to perform remote or distributed builds. But what exactly does this mean? And what are the motivations behind the requests? This post will explore the difference between remote vs. distributed builds and their variations. As there is no industry-wide agreement on consistent terminology for these concepts, the goal of this post is to give an overview of these patterns and how they relate to each other.</p>
<p>Except for two JVM-specific references, these observations are generally applicable to software projects using any language or ecosystem.</p>
<h2 id="but-why">But Why?</h2>
<p>These features are typically discussed in the context of shortening build times on local developer machines. Extended build turnaround times hinder productivity in both local and CI environments, but the local build experience has a disproportionate effect on developer sentiment.</p>
<h2 id="terminology">Terminology</h2>
<p>The terms “remote” and “distributed” builds are not always used consistently in the industry and are often used interchangeably. Below, we’ll give each a distinct definition. Firstly, we’ll also define the more fundamental “build cache” optimization.</p>
<h3 id="what-is-a-remote-build-cache">What is a Remote Build Cache?</h3>
<p>The first pattern that includes a remote component that we’ll discuss is the build cache. Similar to <a href="https://docs.gradle.org/current/userguide/more_about_tasks.html#sec:up_to_date_checks">incremental builds</a>, the build cache avoids execution of CPU-intensive operations like compiling source files or executing tests. While an incremental build leaves the outputs of the most recent local operation in-place on disk, a build cache does this by storing and reusing the results of any previous execution of the operation - much like restoring files from a backup. More importantly, the cache can be local-only, or shared amongst engineers (remote build cache). The CI environment is typically configured to write to a shared cache in the cloud. Each engineer’s machine then pulls results from the shared cache. This means that the same sources need only be built once on a CI host avoiding expensive local compiler invocations on developer machines.</p>
<p>In short, the build cache stores the intermediate build artifacts on remote servers to speed up builds. Unlike remote and distributed builds, no tasks are actually executed remotely.</p>
<p>The Gradle Build Tool <a href="/introducing-gradle-build-cache">Build Cache</a> feature has been very successful at reducing both local and CI build times since its introduction in 2017.</p>
<h3 id="variations-of-remote-build">Variations of Remote Build</h3>
<p>In general, “building remotely” refers to the method of reducing build times by delegating the entire process to another computer. Typically the remote computer is more powerful than a local computer in terms of compute resources and memory. It may host builds in a uniform, curated environment and/or in isolation, free from resource contention with other local processes.</p>
<p>In practice, building remotely can take one of four forms defined below.</p>
<h4 id="legacy-solution-remote-desktopscreen-sharing">Legacy Solution: Remote Desktop/Screen Sharing</h4>
<p>The most primitive pattern of remote building is using remote desktop via the venerable VNC or RDP protocols. While these are admittedly low-tech screen sharing tools and can be greatly affected by low network bandwidth and high latency, they do allow building software on a remote machine. We only mention this historical solution for completeness, as modern remote IDEs provide a much more responsive solution.</p>
<h4 id="what-is-a-remote-build">What is a Remote Build?</h4>
<p>In a remote build scenario, the build command is invoked on the local machine but the actual computation occurs on a remote machine. Source files and other supporting build inputs are initially present on the local machine and synchronized with the remote machine. Likewise, at the conclusion of the build the resulting build outputs/artifacts are synchronized from the remote back to the local machine.</p>
<p>This opens some interesting possibilities, though not without challenges. On the one hand, building remotely may add the ability to build code on a different hardware architecture or operating system (for example, a Windows client builds on a Linux host). And the remote host’s hardware might result in a significant build speed improvement. At the same time, the synchronization overhead of keeping source files, project dependencies, build artifacts and ephemeral, intermediate state of the build software may quickly outweigh the raw speed benefits.</p>
<p>Gradle does not offer its own remote build solution today, but some interesting complementary open-source solutions exist:</p>
<ul>
<li><a href="https://github.com/buildfoundation/mainframer">Mainframer</a>: “A tool that executes a command on a remote machine while syncing files back and forth.”</li>
<li><a href="https://github.com/Adambl4/mirakle">Mirakle</a>: “A Gradle plugin that allows you to move build processes from a local machine to a remote one.”</li>
</ul>
<h4 id="what-is-a-remote-ide">What is a Remote IDE?</h4>
<p>Remote IDE is similar to the remote build scenario, with two key differences. First, the IDE, not a command-line or build tool invocation, handles the communication with the remote host. Next, the source code can be cloned exclusively on the remote host. In this paradigm, the IDE acts as a “thin client” on the local machine. The “backend” portion of the IDE code runs as a background process on the remote host. The benefits of this approach are that the project source code is not required to be present locally, and the synchronization overhead of remote build is alleviated. As with local development, there is potential for resource contention between the “backend” IDE and the build processes.</p>
<p>Three good examples of remote IDEs exist today, all of which work seamlessly with the Gradle Build Tool:</p>
<ul>
<li>Visual Studio Code: Offers a remote IDE experience via its “Remote - SSH” <a href="https://code.visualstudio.com/docs/remote/ssh">extension</a>.</li>
<li>IntelliJ IDEA: JetBrains <a href="https://www.jetbrains.com/help/idea/remote-development-a.html">Client and Gateway</a> work together to run a thin IDE locally while building remotely.</li>
<li>Fleet: Though in closed preview, JetBrains <a href="https://www.jetbrains.com/fleet/">Fleet</a> offers a lightweight remote IDE experience similar to Visual Studio Code.</li>
</ul>
<p>Remote IDEs like the examples above can offer a very pleasant development experience. If the remote host machine or VM is collocated in a data center near the VCS and binary artifact storage systems, cloning code and resolving external dependencies can be extremely quick. If done well, remote IDE can be a substantial improvement compared to building locally.</p>
<h4 id="cloud-development-environment">Cloud Development Environment</h4>
<p>Taking the concept of remote IDE a step further, cloud development environments aim to automate the provisioning of remote hosts, with an emphasis on consistency and collaboration. A cloud development environment is typically centrally-managed, ensuring all engineers have a dependable, uniform environment with no need to build on their local machines. Some cloud development environments combine other dev tooling, such as the IDE, bug/issue tracking and source control. Combined with a remote IDE, wrapping all the tooling an engineer might need into a single, curated environment can yield a very pleasurable and productive development experience.</p>
<p>As with remote build/IDE, additional performance comes in the form of faster CPU cores and improved parallelism via additional cores per machine relative to the local environment. While the use of a cloud development environment does not directly correlate to choice of build tool, the Gradle Build Tool will work transparently in any remote environment.</p>
<p>Three prominent examples of cloud development environments are:</p>
<ul>
<li><a href="https://github.com/features/codespaces">GitHub codespaces</a></li>
<li><a href="https://www.jetbrains.com/remote-development/space-dev-environments/">JetBrains Space</a></li>
<li><a href="https://www.gitpod.io/">Gitpod</a></li>
</ul>
<p>Centrally-managed cloud development environments can provide several benefits:</p>
<ul>
<li>Faster startup time: There is no need for a engineer to manually checkout code and set up the local machine: the environment can be configured to be ready to go “out of the box”.</li>
<li>Faster feedback time: This assumes the remote machine has higher performance, and is colocated with other critical resources such as your binary artifact storage.</li>
<li>Multiplatform support: For example remotely building on a Windows environment from a macOS laptop or vice versa.</li>
<li>Uniform environment: There is less risk of inconsistency on local engineer machines.</li>
<li>Security and audit: This refers to a centrally-managed environment in a data center where it may be desirable for protecting intellectual property or may be a compliance requirement.</li>
</ul>
<h4 id="remote-summary">Remote Summary</h4>
<p>Looking at the available solutions above, we see the most exciting innovation taking place in the remote IDE and cloud development environment spaces. Remote build has some interesting aspects as well, but the marginal benefits to the local developer experience may be outweighed by the increased complexity. As such, we encourage bypassing remote build in favor of remote IDEs while keeping an eye on the emerging capabilities of cloud development environments.</p>
<h3 id="variations-of-distributed-build">Variations of Distributed Build</h3>
<p>Unlike remote builds which execute all work on a single remote machine, distributed builds focus on dividing work into small pieces, and distributing them among multiple machines. The remote executors are typically allocated from a pool, similar to how CI allocation works, though each distributed work item takes relatively little time to execute.</p>
<p>Distributed builds are implemented as a more-or-less transparent feature of the build, so developers trigger them locally similarly to how they’d trigger a local build. Inputs that are required to execute a work item are transmitted to the executor, and generated outputs are synced back.</p>
<p>Don’t forget about the backing infrastructure needs of distributed builds. A build could actually be slower if sufficient remote build agents are not available. The management of a complex pool of build distribution agents adds additional maintenance like monitoring/observability, scaling and failover/fault tolerance.</p>
<p>Before we get into true distributed build solutions, we’ll first describe the most basic technique for distributing build work across multiple machines.</p>
<h4 id="manual-optimization-ci-fanout">Manual Optimization: CI Fanout</h4>
<p>CI fanout is a technique to reduce end-to-end build time by splitting the build (typically subsets of tests) to multiple CI jobs so that the work is executed on different agents. While it is an improvement over no parallelism or single-machine parallelism, it comes with major drawbacks. The partitioning of the CI jobs must be manually configured, and is unique to each CI platform. While this reduces overall build times on CI, it does not benefit local builds. Other challenges of this approach are described <a href="https://gradle.com/gradle-enterprise-solutions/test-distribution/#advantages-section">here</a>.</p>
<h4 id="modern-test-distribution">Modern Test Distribution</h4>
<p>In our experience, running tests, not compiling source code, is typically the primary cause of slow builds, especially in the JVM ecosystem. Test execution on the JVM is naturally suited for distribution, as tests are typically executed in a separate ephemeral VM whose system environment, classpath and memory usage are already specified. These parameters are easily communicated to a remote host for distributed execution.</p>
<p>The same concerns when locally partitioning or parallelizing test execution apply when running tests in a distributed fashion. Good test methods should be atomic, relying only on explicit environment setup/teardown instructions from test fixtures. A poorly-crafted, non-atomic test relying on side-effects of another test may fail in unexpected ways when executed in distributed fashion.</p>
<p>Gradle Enterprise’s <a href="https://gradle.com/gradle-enterprise-solutions/test-distribution/">Test Distribution</a> commercial feature executes tests on a pool of remote hosts with greater parallelism than can be achieved locally. It also executes all methods of a test class on the same host, alleviating the most common cause of non-atomic test failures mentioned above.</p>
<h4 id="general-distribution">General Distribution</h4>
<p>As the name suggests, general distribution is a way to execute any build operation on a remote host. Careful consideration must be given to environment variables or other system attributes which could produce unintended changes for a distributed build result compared to building the same code locally. Also, the question of which build operations justify the overhead of distribution is difficult to answer.</p>
<p>The following tools take a general approach to build distribution:</p>
<ul>
<li>Pants: <a href="https://www.pantsbuild.org/docs/remote-execution">https://www.pantsbuild.org/docs/remote-execution</a></li>
<li>Bazel: <a href="https://bazel.build/docs/remote-execution">https://bazel.build/docs/remote-execution</a></li>
</ul>
<p>Before deciding on a general distribution solution, know that significant tradeoffs may be required. A general distribution build environment may add significant complexity to the build logic. “Split package” compilation (sometimes called the <a href="https://v1.pantsbuild.org/build_files.html#target-granularity">1:1:1 rule</a>) is a technique to divide source code into smaller units of compilation to aid in distributability, but adds more pain to the already complex issue of dependency management. See <a href="/gradle-vs-bazel-jvm#the-granularity-of-build-files">The granularity of build files</a> for more detail.</p>
<p>The benefit of distributing non-test work such as compilation depends on the compilation speed of programming language in use. For example, Java compilation is relatively fast compared to “native” languages. Small performance improvements might be possible using “split package” compilation, mentioned above. But the additional pain of maintaining complex build logic may not be justifiable for relatively minor performance gains. This is especially true considering the Gradle Build Tool’s incremental compiler for Java already provides a significant performance boost on top of javac.</p>
<p>See the <a href="/general-build-distribution">General Build Distribution: A Game-Changer or a Gimmick?</a> article for more details about the tradeoffs with general build distribution.</p>
<h3 id="common-factors-of-remote-and-distributed-builds">Common Factors of Remote and Distributed Builds</h3>
<p>In both the remote and distributed paradigms, non-trivial amounts of network traffic can result. Serializing source code to a remote host or synchronizing build artifacts between agents can incur a significant overhead. Network proximity between the local client and remote host, or between distribution agents and artifact storage, can be a major factor. Network connections should be both high-bandwidth and low-latency to prevent eroding the theoretical gains achieved through remote and/or distributed work.</p>
<p>Further, managing the pool of remote hosts or distributed agents incurs more cost and overhead. Additional engineering investment will be required to provide the standardized environments. Care should be exercised to avoid resource starvation or over-allocation by responding to peak usage cycles and downtime.</p>
<h2 id="summary">Summary</h2>
<p>In this post, we’ve reviewed the build patterns in which remote machines are leveraged, clarified the definition of remote and distributed builds, and discussed their variations.</p>
<p>We started by explaining the remote build cache as the most fundamental build optimization leveraging remote machines. Then, we elaborated on remote build patterns and pointed out exciting innovation taking place in the remote IDE and cloud development environment spaces. Finally, we explained the CI fanout technique and different patterns in distributed builds including test distribution and general distribution.</p>
<h2 id="feedback">Feedback</h2>
<p>Let us know if you have any questions on our <a href="https://discuss.gradle.org/">forums</a> or <a href="https://gradle.com/slack-invite">Gradle Community Slack</a>.</p>
Mentoring Program Announcement2022-06-23T00:00:00-04:00https://blog.gradle.org/mentoring-program-annoucementAmanda Martin
<p>The lack of diversity in the technical world is a chronic, pervasive issue. For example, a <a href="https://www.statista.com/statistics/1126823/worldwide-developer-gender/#:~:text=According%20to%20a%20global%20software,of%20the%20software%20development%20job.">2021 global software developer survey</a> showed that female software developers are only 5 percent of the population. This is something we need to change.</p>
<p>To help promote diversity in our industry, Gradle is excited to announce a new mentorship program. The program is open to everyone, but we’ll be prioritizing mentees from <a href="https://services.google.com/fh/files/misc/diversity-gaps-in-computer-science-report.pdf">underrepresented segments of the developer community, including women, and Black and Hispanic</a> individuals, who want support and enhanced professional development from members of the Gradle Build Tool community. Mentees in the program will have access to personalized discussions on using Gradle Build Tool, build automation, developer productivity and more.</p>
<p>Members of our <a href="https://blog.gradle.org/gradle-fellowship">Gradle Fellowship</a> are volunteering their time towards establishing a diverse developer landscape. The Gradle Fellows will be matched with more junior community members (mentees) seeking to learn and enhance skills that will help further their career.</p>
<p>If you are new to Gradle Build Tool and looking for growth, submit <a href="https://forms.gle/b1Srxr1MyAhJvTvs9">your application</a> or share our program with others in need.</p>
<p>—</p>
<h2 id="feedback">Feedback</h2>
<p>Let us know if you have any questions on our <a href="https://discuss.gradle.org/">forums</a> or <a href="https://gradle.com/slack-invite">Gradle Community Slack</a>.</p>
Gradle Public Roadmap2022-06-17T00:00:00-04:00https://blog.gradle.org/roadmap-announcementPiotr Jagielski
<p>We are excited to announce that the Gradle Build Tool roadmap is now public. See the <a href="https://github.com/orgs/gradle/projects/31/views/1">roadmap board</a> and the associated <a href="https://github.com/gradle/build-tool-roadmap">readme</a>.</p>
<h2 id="motivation">Motivation</h2>
<p>The Gradle Build Tool project regularly ships new releases full of new features and bug fixes. So far our main channels of communication with the user community to share plans and get feedback were <a href="https://gradle.org/releases/">release notes</a>, <a href="https://blog.gradle.org/">blog</a>, <a href="https://github.com/gradle/gradle/issues">issue tracker</a>, <a href="https://discuss.gradle.org/">forums</a>, and <a href="https://gradle-community.slack.com">community Slack</a>. However, we lacked a high-level overview of what’s planned in the upcoming releases.</p>
<h2 id="about-the-roadmap">About the roadmap</h2>
<p>Today, we are announcing our new <a href="https://github.com/orgs/gradle/projects/31/views/1">roadmap board</a> hosted on GitHub. It lists major projects and user-facing features, organized by quarter and target release.</p>
<p>The board also shows the past two quarters so that you can see what we delivered in recent Gradle releases.</p>
<p><img src="https://user-images.githubusercontent.com/51727488/174340994-e2e84635-acc0-4122-b5fd-cf6bd69fd923.png" alt="roadmap-board" /></p>
<p>Each roadmap item provides a brief description and links to related issues in the <a href="https://github.com/gradle/gradle/issues">issue tracker</a>.</p>
<p><img src="https://user-images.githubusercontent.com/51727488/174341008-90d041a3-3ad8-4221-b66d-cf6057a434cb.png" alt="roadmap-tracker" /></p>
<h2 id="share-your-feedback">Share Your Feedback</h2>
<p>We hope that this additional level of transparency about our plans will be useful to you. Please let us know if you have any questions or feedback.</p>
<p>Comments on the roadmap board itself are disabled to keep it neat and tidy. Instead, reach out to us on the <a href="https://gradle-community.slack.com">community Slack</a> #roadmap channel.</p>
Gradle Plugin Resolution Outage Postmortem2022-01-20T00:00:00-05:00https://blog.gradle.org/plugins-jcenterLouis Jacomet
<p>On January 12th 2022, Gradle users were experiencing issues resolving plugins from the <a href="https://plugins.gradle.org">Gradle Plugin Portal</a> because of the outage of JCenter that the Plugin Portal depends on for some of the functionality.</p>
<p>This postmortem gives a timeline of the outage, describes the effect on Gradle users, and what actions were taken to further reduce the Gradle Plugin Portal’s reliance on JCenter.</p>
<p>Finally, we will also discuss how <em>you</em> can protect <em>your build</em> against such outages.</p>
<p>Users affected by JCenter outages for project dependencies should also refer to <a href="/jcenter-shutdown">our original blog post</a> about the shutdown of JCenter.</p>
<h2 id="outage-timeline">Outage timeline</h2>
<dl>
<dt>4.30pm UTC, January 12th <em>Outage begins</em></dt>
<dd>Gradle users start noticing that plugin resolution is failing in builds.</dd>
<dt>4.56pm UTC, January 12th <em>Incident opened</em></dt>
<dd>The outage is acknowledged on the <a href="https://status.gradle.com">Gradle status page</a>.</dd>
<dt>6pm UTC, January 12th</dt>
<dd>We identify a hotfix that will allow the Gradle Plugin Portal to ignore <code class="language-plaintext highlighter-rouge">5xx</code> error codes from JCenter and resolve plugins it hosts entirely.
However, the hotfix cannot be built and deployed immediately because building the Plugin Portal code requires resolving some plugins from the Plugin Portal itself and JCenter, which failed.</dd>
<dt>8.44pm UTC, January 12th <em>First fix is deployed</em></dt>
<dd>We update the build of the Plugin Portal to not rely on JCenter at all by using <a href="https://docs.gradle.org/7.3.3/userguide/declaring_repositories.html#sec:repository-content-filtering">repository content filtering</a> and <a href="https://docs.gradle.org/7.3.3/userguide/declaring_repositories.html#sec:supported_metadata_sources">supported metadata sources</a>.
The first fix is deployed.</dd>
<dt>8.58pm UTC, January 12th <em>Second fix is deployed</em></dt>
<dd>The first fix turns out to be insufficient.
We develop and deploy a second fix that allows us to change the repository to which the Plugin Portal redirects.
We configure the Plugin Portal to use our internal mirror of JCenter.</dd>
<dt>9.43pm UTC, January 12th <em>Outage is resolved</em></dt>
<dd>The existing mirror is too sparsely populated as it was not used much, which means most builds still fail to resolve plugins.
We update the mirror repository to redirect to Maven Central as well.
This results in most dependencies being properly resolved and unblocked most builds.
Plugins that rely on artifacts that were only available on JCenter still fail to be resolved.</dd>
<dt>6.35am UTC, January 13th <em>Incident is closed</em></dt>
<dd>JCenter is back online.
The mirror repository is changed back to mirror JCenter only and the Gradle Plugin Portal remains configured to use the mirror repository.</dd>
</dl>
<h2 id="effects-of-outage-on-builds">Effects of outage on builds</h2>
<p>This outage showed a few ways that builds may fail when the Plugin Portal cannot serve all requests:</p>
<ol>
<li>Builds that use dynamic versions for plugins could not resolve the list of versions.</li>
<li>Builds that run on ephemeral CI agents or in new environments could download plugin artifacts, but they could not download dependencies for the plugins from JCenter.</li>
<li>Plugins that missed artifacts requested by Gradle on the Plugin Portal failed to resolve because JCenter would serve <code class="language-plaintext highlighter-rouge">5xx</code> errors instead of <code class="language-plaintext highlighter-rouge">404</code>, which Gradle considers an error instead of a missing artifact.</li>
</ol>
<h2 id="reducing-the-chance-of-future-outages">Reducing the chance of future outages</h2>
<p>Issues with resolving Gradle plugins have a high impact on Gradle builds.
Below we describe the steps we are taking to reduce the likelihood and severity of potential future outages.</p>
<h3 id="immediate-changes">Immediate changes</h3>
<p>Following this incident, the Gradle Plugin Portal now uses a JCenter mirror hosted by Gradle instead of JCenter directly.
This should shield users from short JCenter outages for libraries that have been cached by the mirror.
We saw another short outage of JCenter over the weekend and this did not appear to impact Gradle Plugin Portal users.</p>
<h3 id="background-information">Background information</h3>
<p>The link between the Gradle Plugin Portal and JCenter is largely historical.</p>
<p>Before the announcement of <a href="https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/">the sunset of Bintray and JCenter</a>, you could publish a Gradle plugin in two different ways:</p>
<ul>
<li>To a Bintray repository.
Plugin authors could then ask for the plugin to be included on the Plugin Portal.</li>
<li>To the Plugin Portal.
Plugin authors could use <a href="https://plugins.gradle.org/docs/publish-plugin">the <code class="language-plaintext highlighter-rouge">plugin-publish</code> plugin</a> to upload their plugins directly.</li>
</ul>
<p>The Gradle Plugin Portal relies on the following from JCenter:</p>
<ol>
<li>List of available versions: Plugins that were published to Bintray and then on the Plugin Portal still need to expose <em>all</em> of their versions to plugin users.</li>
<li>Missing artifacts: Files like <a href="https://docs.gradle.org/7.3.3/userguide/publishing_gradle_module_metadata.html">Gradle Module Metadata</a>, checksums and signature files were not always supported by the Plugin Portal.
These files may have been published directly to JCenter and not the Plugin Portal.</li>
<li>Plugin dependencies: Dependencies used by a plugin, like Guava, are not deployed to the Plugin Portal but need to be reachable as if they are, which is what the redirect offers.</li>
</ol>
<p>We discontinued and removed the integration with Bintray when Bintray shutdown in May 2021, but we kept redirecting to JCenter since JFrog said that JCenter would remain read-only indefinitely.
We are now reconsidering that decision and will move to reduce and eliminate the Plugin Portal’s dependency on JCenter.</p>
<p>The Gradle team is aiming to find a solution that will have as little impact as possible on builds, but we know that zero impact will not be possible.</p>
<p>We will provide a future update on how we will be replacing JCenter on the Plugin Portal and how that may affect your builds.</p>
<h2 id="strengthening-your-build-infrastructure">Strengthening your build infrastructure</h2>
<p>Build engineers can strengthen their build infrastructure by decoupling it from online services like JCenter and the Gradle Plugin Portal.</p>
<p>By setting up a <a href="https://plugins.gradle.org/docs/mirroring">mirror of the Plugin Portal</a>, you can cache artifacts closer to your build infrastructure and developers.</p>
<p>This can provide multiple benefits:</p>
<ul>
<li>Improved performance</li>
<li>Better control of used dependencies for security and auditing purposes</li>
<li>Isolation from outages</li>
</ul>
<p>If you’re already using custom repositories, you can also leverage <a href="https://docs.gradle.org/7.3.3/userguide/declaring_repositories.html#sec:repository-content-filtering">repository content filtering</a> to make sure that you fetch build and project dependencies from a particular repository.</p>
<h2 id="feedback">Feedback</h2>
<p>Let us know if you have any questions on our <a href="https://discuss.gradle.org/">forums</a> or <a href="https://gradle.com/slack-invite">Gradle Community Slack</a>.</p>
Dealing with the Critical Log4j Vulnerability2021-12-13T00:00:00-05:00https://blog.gradle.org/log4j-vulnerabilityKyle Moore
<p>A critical remote code execution (RCE) vulnerability has been identified in the popular Apache Log4j logging library that affects versions 2.0 up to and including 2.14.1. This vulnerability has affected a very large number of JVM-based systems. For more information on the vulnerability itself, see <a href="https://nvd.nist.gov/vuln/detail/CVE-2021-44228">CVE-2021-44228</a>.</p>
<p><strong>Update (December 22, 2021)</strong>: Since the first post, two other vulnerabilities have been identified - <a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046">CVE-2021-45046</a> and <a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105">CVE-2021-45105</a> - so make sure to go over the different sections for updated instructions.</p>
<p>This vulnerability is being actively exploited. All Gradle users should assess whether their software projects are vulnerable and, if necessary, update to <a href="https://logging.apache.org/log4j/2.x/changes-report.html#a2.17.0">Log4j 2.17.0</a> or newer as soon as possible. We have <a href="#protecting">provided instructions below</a> on how to identify and prevent this vulnerability in your project.</p>
<p>We strongly recommended that you configure your Gradle build to reject any vulnerable version of Log4j using a <a href="https://docs.gradle.org/current/userguide/rich_versions.html#rich-version-constraints">dependency constraint</a>.</p>
<p>In addition to your project dependencies, we also recommend protecting your build dependencies as documented below.</p>
<p>Note that the Gradle Build Tool itself is <em>not</em> impacted by this vulnerability as it does not use Log4j. Gradle uses SLF4J and a custom logging implementation not susceptible to the vulnerable string substitution.</p>
<p>The <a href="https://docs.gradle.org/current/userguide/scala_plugin.html">Gradle Scala plugin</a> uses the Zinc Scala compiler that has a dependency on a vulnerable version of Log4j. However, in this case Gradle also supplies its own logging implementation and Log4j is not used by default.</p>
<blockquote>
<p>Updates to this post:</p>
<ul>
<li><a href="#2-upgrade-the-log4j-dependency-to-a-non-vulnerable-version">Updated</a> on December 14th</li>
<li><a href="#protecting-your-build-dependencies">Updated</a> on December 15th</li>
<li><a href="#2-upgrade-the-log4j-dependency-to-a-non-vulnerable-version">Updated</a> on December 22nd</li>
<li><a href="#protecting-plugin-portal-users">Updated</a> on December 30th</li>
</ul>
</blockquote>
<p><a name="protecting"></a></p>
<h2 id="protecting-your-project-dependencies">Protecting your project dependencies</h2>
<h3 id="1-identify-if-your-project-uses-a-vulnerable-log4j-version">1. Identify if your project uses a vulnerable Log4j version</h3>
<p>First, verify if your project uses the vulnerable Log4j version using the <a href="https://docs.gradle.org/current/userguide/viewing_debugging_dependencies.html#sec:listing_dependencies">dependencies report</a> or a <a href="https://scans.gradle.com/">Build Scan™</a>. See <a href="https://docs.gradle.org/current/userguide/viewing_debugging_dependencies.html#sec:identifying_reason_dependency_selection">viewing and debugging dependencies</a> for details.</p>
<p>All versions of <code class="language-plaintext highlighter-rouge">org.apache.logging.log4j:log4j-core</code> between 2.0 and 2.16.0 (inclusive) are vulnerable.</p>
<h3 id="2-upgrade-the-log4j-dependency-to-a-non-vulnerable-version">2. Upgrade the Log4j dependency to a non-vulnerable version</h3>
<p>Upgrade the Log4j dependency in the <code class="language-plaintext highlighter-rouge">dependencies</code> block of the build scripts for each subproject or in your <a href="https://docs.gradle.org/current/userguide/platforms.html#sub:version-catalog">version catalog</a> in the case you are using central declaration of dependencies.</p>
<p><strong>Update (December 14, 2021)</strong>: Log4j released <a href="https://logging.apache.org/log4j/2.x/changes-report.html#a2.16.0">2.16.0</a> which disables and removes support for the feature at the heart of the vulnerability. In some non-default configurations, software <a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046">could still be vulnerable with 2.15.0</a>. To be absolutely certain, you should update to 2.16.0. The instructions below have been modified to use 2.16.0.</p>
<p><strong>Update (December 22, 2021)</strong>: Log4j released <a href="https://logging.apache.org/log4j/2.x/changes-report.html#a2.17.0">2.17.0</a> which fixes <a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105">yet another vulnerability</a>. To be absolutely certain, you should update to 2.17.0. The instructions below have been modified to use 2.17.0.</p>
<p><a name="constraint"></a></p>
<h3 id="3-prevent-accidental-resolution-of-a-vulnerable-log4j-version-in-your-project">3. Prevent accidental resolution of a vulnerable Log4j version in your project</h3>
<p>Given the critical severity of this vulnerability, we recommend taking the additional step described below in order to prevent accidental inclusion of a vulnerable version. Note that even if you are not using Log4j directly, the vulnerable version may still be resolved transitively through one your dependencies.</p>
<p>Use the <a href="https://docs.gradle.org/current/userguide/rich_versions.html#rich-version-constraints">dependency constraints</a> feature to make sure that your project cannot resolve an impacted Log4j version by adding the following snippet to your Gradle build:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dependencies</span> <span class="o">{</span>
<span class="n">constraints</span> <span class="o">{</span>
<span class="n">implementation</span><span class="o">(</span><span class="s2">"org.apache.logging.log4j:log4j-core"</span><span class="o">)</span> <span class="o">{</span>
<span class="n">version</span> <span class="o">{</span>
<span class="n">strictly</span><span class="o">(</span><span class="s2">"[2.17, 3["</span><span class="o">)</span>
<span class="n">prefer</span><span class="o">(</span><span class="s2">"2.17.0"</span><span class="o">)</span>
<span class="o">}</span>
<span class="n">because</span><span class="o">(</span><span class="s2">"CVE-2021-44228, CVE-2021-45046, CVE-2021-45105: Log4j vulnerable to remote code execution and other critical security vulnerabilities"</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The constraint will not be activated unless log4j-core appears in the dependency graph. The constraint will either forcefully upgrade the dependency to at least 2.17.0 or fail the build if the strict version constraint cannot be satisfied.</p>
<p>Note that using the <code class="language-plaintext highlighter-rouge">implementation</code> configuration here will cover the default setup of a Java library or application. But if you have custom configurations or more advanced setup, you should also add this constraint to additional configurations.</p>
<p>In addition, if you publish your library with such a constraint defined with <a href="https://docs.gradle.org/current/userguide/publishing_gradle_module_metadata.html">Gradle Module Metadata</a>, it will also cause any builds consuming your library to fail if they attempt to resolve a vulnerable version of log4j-core.</p>
<h3 id="4-prevent-using-a-vulnerable-log4j-version-organization-wide">4. Prevent using a vulnerable Log4j version organization-wide</h3>
<p>Your organization may leverage shared plugins that are applied to all projects. In this case, you can also apply the constraint described above to all the projects in the organization.</p>
<p>If you use <a href="https://docs.gradle.org/current/userguide/custom_plugins.html#sec:precompiled_plugins">precompiled script plugins</a>, simply copy the above <a href="#constraint">snippet</a> to the script applied to all your JVM projects. If you use binary plugins, here is the Java code equivalent to the DSL snippet:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">project</span><span class="o">.</span><span class="na">getDependencies</span><span class="o">().</span><span class="na">constraints</span><span class="o">(</span><span class="n">constraints</span> <span class="o">-></span> <span class="o">{</span>
<span class="n">constraints</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="s">"implementation"</span><span class="o">,</span> <span class="s">"org.apache.logging.log4j:log4j-core"</span><span class="o">,</span> <span class="n">c</span> <span class="o">-></span> <span class="o">{</span>
<span class="n">c</span><span class="o">.</span><span class="na">version</span><span class="o">(</span><span class="n">v</span> <span class="o">-></span> <span class="o">{</span>
<span class="n">v</span><span class="o">.</span><span class="na">strictly</span><span class="o">(</span><span class="s">"[2.17, 3["</span><span class="o">);</span>
<span class="n">v</span><span class="o">.</span><span class="na">prefer</span><span class="o">(</span><span class="s">"2.17.0"</span><span class="o">);</span>
<span class="o">});</span>
<span class="n">c</span><span class="o">.</span><span class="na">because</span><span class="o">(</span><span class="s">"CVE-2021-44228, CVE-2021-45046, CVE-2021-45105: Log4j vulnerable to remote code execution and other critical security vulnerabilities"</span><span class="o">);</span>
<span class="o">});</span>
<span class="o">});</span>
</code></pre></div></div>
<h2 id="protecting-your-build-dependencies">Protecting your build dependencies</h2>
<p><del><strong>Update (December 15, 2021)</strong>: You do not need to apply this to Gradle 7.3.2 and above. Gradle automatically requires the version of Log4J to be 2.16.0 or higher.</del></p>
<p><strong>Update (December 22, 2021)</strong>: You do not need to apply this to Gradle 7.3.3 and above. Gradle automatically requires the version of Log4J to be 2.17.0 or higher. Gradle 6.9.2 also does this for users still on the 6.x line.</p>
<p>In addition to project dependencies, your project probably uses either internal or third-party Gradle plugins. These plugins also have their own dependencies that may accidentally bring a vulnerable Log4j dependency to the build classpath.</p>
<p>In other words, your build logic may be vulnerable in the same way that your production code. While such vulnerabilities are likely harder to exploit, you can also protect your build dependencies with the snippet below:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">buildscript</span> <span class="o">{</span>
<span class="n">dependencies</span> <span class="o">{</span>
<span class="n">constraints</span> <span class="o">{</span>
<span class="n">classpath</span><span class="o">(</span><span class="s2">"org.apache.logging.log4j:log4j-core"</span><span class="o">)</span> <span class="o">{</span>
<span class="n">version</span> <span class="o">{</span>
<span class="n">strictly</span><span class="o">(</span><span class="s2">"[2.17, 3["</span><span class="o">)</span>
<span class="n">prefer</span><span class="o">(</span><span class="s2">"2.17.0"</span><span class="o">)</span>
<span class="o">}</span>
<span class="n">because</span><span class="o">(</span><span class="s2">"CVE-2021-44228, CVE-2021-45046, CVE-2021-45105: Log4j vulnerable to remote code execution and other critical security vulnerabilities"</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The snippet should be applied to the <code class="language-plaintext highlighter-rouge">buildscript</code> block in each build script and also to the settings.gradle(.kts) file, and ensures only Log4j 2.17.0 and above are resolvable as build dependencies. The statement must be at the top of the file.</p>
<h2 id="protecting-plugin-portal-users">Protecting Plugin Portal users</h2>
<p>Given the severity of the initial Log4j vulnerability, the Gradle team wanted to improve the situation for users of the Plugin Portal.
We are taking steps to make sure that no new plugin can have a dependency on a vulnerable Log4j version.
The following changes are available already and will be made mandatory in a later update.</p>
<p>As a Gradle plugin author, it is possible that you have either direct, or transitive dependencies on the vulnerable Log4j versions.
In order to protect your plugin users, Gradle has added new validation rules to its <code class="language-plaintext highlighter-rouge">plugin-publish</code> plugin.</p>
<p>We have released <a href="https://plugins.gradle.org/plugin/com.gradle.plugin-publish/0.19.0">version 0.19.0 of the Plugin Publish Plugin</a> which automatically detects vulnerable Log4j plugin dependencies and, if any are found, blocks the plugin publication by failing the build.
There is no automated upgrade as normally a Gradle plugin does not need Log4j since the Gradle runtime does not do its logging with this library.
It is the job of the plugin author to fix the problem by removing the dependency or upgrading it.</p>
<p>Using this version of the Plugin Publish Plugin will become mandatory in the near future, by having the Plugin Portal reject any publications done with older versions of the plugin.</p>
<h2 id="gradle-enterprise-products">Gradle Enterprise products</h2>
<p>Several Gradle Enterprise products <a href="https://security.gradle.com/advisory/2021-11">were impacted</a> by the Log4j vulnerability.</p>
<p>New versions of Gradle Enterprise, Test Distribution Agent and Build Cache Node were released December 13, 2021.</p>
<p><strong>Update (December 22, 2021)</strong>: Gradle Enterprise 2021.4 upgrades Log4j to 2.17.0</p>
<h2 id="feedback">Feedback</h2>
<p>Let us know if you have any questions on our <a href="https://discuss.gradle.org/">forums</a> or <a href="https://gradle.com/slack-invite">Gradle Community Slack</a>.</p>
Automatic Feedback on Plugin Publication2021-11-04T00:00:00-04:00https://blog.gradle.org/plugin-portal-validationsJozsef Bartok
<p>New Gradle plugins undergo a manual inspection before they get approved to be published in the <a href="https://plugins.gradle.org/">Plugin Portal</a>.
Most of the performed checks are now fully automated in order to reduce wait time for plugin authors and reduce the
risk of human errors. This blog post describes the new automation affecting community plugin authors and explains
that it is important for the security of the ecosystem.</p>
<h1 id="plugin-publishing-security">Plugin Publishing Security</h1>
<p>One of our priorities at Gradle is reducing the risk of supply chain attacks. Such insidious attacks can have a
very large scale and impact. For example, a malicious community plugin may potentially infect a large number of
software systems built with Gradle. You can read more about such attacks and some examples in the <a href="#references">following blog posts</a>.</p>
<p>During the past year, we at Gradle have been gradually ramping up our efforts to mitigate the risks associated with
supply chain attacks. In the context of the Plugin Portal, our main objectives were:</p>
<ul>
<li>stronger enforcement of plugin domain ownership</li>
<li>preventing plugin code dependencies from shadowing other, “real” dependencies of builds</li>
</ul>
<p>When a new plugin is <a href="https://plugins.gradle.org/docs/publish-plugin">published on the Gradle Plugin Portal</a> it goes
through a manual review process. This usually happens only for the initial version. A Gradle engineer inspects
the metadata and decides whether to publish it to the portal or request changes to it.</p>
<p>To give you a better understanding of what kind of sanity checks the engineer is doing, here are some examples:</p>
<ul>
<li>Is the plugin being published to misleading GAV (group, artifact, version) coordinates?</li>
<li>Does the publishing user belong to the organization owning the plugin?</li>
<li>Does the plugin have a clear and useful description?</li>
<li>Does the plugin have the right tags?</li>
<li>Is the link to the documentation valid and accessible?</li>
</ul>
<p>Even though this approach has been working pretty well, it unnecessarily increases the time it takes to provide feedback
to plugin authors, either to fix problems they might have or to get their plugins published on the Portal. The manual
aspect also meant that human errors were still possible.</p>
<h1 id="full-automation">Full Automation</h1>
<p>We have decided to automate most of the approval checks as part of the <a href="https://plugins.gradle.org/docs/publish-plugin">plugin publication process</a>.
Publishing will now fail if any of the checks are unsuccessful and the approval process won’t even start. This provides
immediate feedback to the plugin author if any obvious issues are present, instead of the current need to spend time
waiting upon a manual review (and rejection).</p>
<p>Most of the checks are run on the server side, so the set of conditions that can be verified is pretty extensive.
One of the effects of server-side checking is that these checks will be performed regardless of which version of
the <a href="https://plugins.gradle.org/plugin/com.gradle.plugin-publish">Plugin Publishing Plugin</a> the author is using.</p>
<p>We understand that having publications refused by automatic checks might create a bit of frustration at first, but
we would like to emphasize that the criteria for accepting plugins remain the same. Any plugin automatically
rejected now would have been refused anyway during the previous manual approval process but after a longer delay.
By providing immediate feedback, plugin authors can fix issues with plugin approval while the code is still fresh
in the heads, instead of picking things up again later.</p>
<p>Moreover, the checking process attempts to provide comprehensive messages about all the problems it finds, not just the
first one it comes across, so the number of retries should be minimal.</p>
<p>We have also added a <a href="https://plugins.gradle.org/docs/publish-plugin#troubleshooting">Troubleshooting Guide</a> to the
plugin publication documentation, to help solve the most frequent problems.</p>
<h1 id="exceptions">Exceptions</h1>
<p>While the automation is extensive, not all plugin approval checks are safe to do in this manner. Some things will always
require human judgment. We hope that, even though these changes do not remove the need for a manual approval
process, they should greatly increase the number of plugins that can be approved without requiring further change.
This should help plugin authors successfully publish much more quickly.</p>
<p>One further thing to note is that most of the automated checks still only run on the initial version of plugins, so
publishing updates to plugins should remain as simple as it has been until now.</p>
<h1 id="references">References</h1>
<ul>
<li><a href="https://www.wired.com/story/hacker-lexicon-what-is-a-supply-chain-attack/">Hacker Lexicon: What Is a Supply Chain Attack?</a></li>
<li><a href="https://jfrog.com/blog/yet-another-case-for-using-exclude-patterns-in-remote-repositories/">Yet Another Case for Using Exclude Patterns in Remote Repositories: Namespace Shadowing (a.k.a. “Dependency Confusion”) Attack</a></li>
<li><a href="https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610">Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies - The Story of a Novel Supply Chain Attack</a></li>
</ul>
JCenter Shutdown Impact on Gradle Builds2021-02-22T00:00:00-05:00https://blog.gradle.org/jcenter-shutdownSterling Greene
<p>On February 3 2021, JFrog <a href="https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/">announced that they will be shutting down Bintray and JCenter</a>. This post tells you what you need to know and do to avoid disruptions to your build pipelines.</p>
<p>Your build may be affected by this shutdown in several ways:</p>
<ul>
<li><a href="#impact-builds">Gradle may not be able to download the dependencies used to compile, test or run your code.</a></li>
<li><a href="#impact-plugins">Gradle may not be able to download the dependencies used by plugins to configure your build.</a></li>
<li><a href="#impact-publishing">Gradle may no longer be able to publish your package to Bintray.</a></li>
</ul>
<p>Additionally, you should be aware of the <a href="#security-considerations">security considerations</a> when moving from one repository to another.</p>
<p><strong>UPDATE</strong>: JFrog has decided to keep JCenter as a read-only repository indefinitely. New package and versions are no longer accepted on JCenter. All Bintray services have been shutdown.</p>
<h2 id="background">Background</h2>
<p>JCenter is a central artifact repository, like <a href="https://search.maven.org/">Maven Central</a>. Software projects use JCenter to distribute their software to other people. JCenter also serves as a mirror for Maven Central, so any dependencies available on Maven Central are also available on JCenter (but not vice versa).</p>
<p>Bintray is a management layer that software projects use to publish and promote packages to JCenter. Bintray also allowed users to create public user-specific repositories that were isolated from JCenter.</p>
<p>Both of these services are affected by the shutdown.</p>
<p><a name="impact-builds"></a></p>
<h2 id="impact-to-builds">Impact to builds</h2>
<p>By default, Gradle does not add an artifact repository to your project; however, Gradle does provide a <a href="https://docs.gradle.org/current/userguide/declaring_repositories.html#sub:maven_jcenter">convenient API</a> for using the JCenter repository.</p>
<p>The Gradle <a href="https://docs.gradle.org/current/userguide/build_init_plugin.html#header">Build Init plugin</a> produces build templates that use the JCenter repository to resolve dependencies. In many code examples and documentation, JCenter is used as an example repository.</p>
<p>It’s very likely that you have builds that rely on JCenter. We’ve found over a million Git repositories on GitHub that use JCenter with Gradle.</p>
<p>To discourage new projects from using JCenter, we will be removing JCenter from our samples and init templates. The new default will be Maven Central. Gradle itself has no inherent tie to JCenter or Maven Central, so you can always switch any other repository of your choice. This change will be effective with the next Gradle release – Gradle 7.0.</p>
<p>Gradle 7.0 will also deprecate the use of <code class="language-plaintext highlighter-rouge">jcenter()</code> to resolve dependencies. You will still be able to use JCenter as a repository, but Gradle will emit a deprecation warning. The <code class="language-plaintext highlighter-rouge">jcenter()</code> method will be removed in the next major release.</p>
<p><img src="/images/jcenter-shutdown/deprecation-build-scan.png" alt="Example of deprecation message in a build scan" /></p>
<p>Packages that are hosted on JCenter will need to find an alternative repository to provide updates after March 31 2021. Many packages will likely migrate to Maven Central, but some packages may not migrate at all and some may only publish new versions. Abandoned projects could change their coordinates as other maintainers take over. This means the transition from JCenter to another repository will be more difficult than simply using a different repository URL.</p>
<p>Based on the current timeline, builds that use JCenter will be able to resolve dependencies until February 1, 2022 without changes. After that date, there are no guarantees that you will be able to build your software if you continue to use JCenter.</p>
<h3 id="what-do-you-do">What do you do?</h3>
<p>Follow the steps below to prepare your build for the JCenter shutdown:</p>
<ol>
<li><a href="#check">Determine if your build uses JCenter or not</a></li>
<li>Remove JCenter from your build and replace it with Maven Central. Gradle provides a <a href="https://docs.gradle.org/current/userguide/declaring_repositories.html#sub:maven_central">convenient API for that repository too</a>.</li>
<li>Run your build pipeline to see if everything works still.
<ul>
<li>If your build is successful, you’re done.</li>
<li>If your build fails, you’ll need to <a href="#troubleshooting">troubleshoot which dependencies still require JCenter</a>.</li>
</ul>
</li>
</ol>
<p><a name="check"></a></p>
<h3 id="how-do-you-know-if-youre-using-jcenter">How do you know if you’re using JCenter?</h3>
<p>There are a few different ways you can check if you’re using JCenter.</p>
<p>JCenter has been the default repository for Android projects for many years. If you’re building an Android application, you’re probably using JCenter.</p>
<p>The Gradle Plugin Portal implicitly mirrors JCenter currently. If you’re using the Plugin Portal (via <code class="language-plaintext highlighter-rouge">gradlePluginPortal()</code> or the URL plugins.gradle.org/m2) to resolve your application’s dependencies, you may be relying on JCenter. You should avoid using the Plugin Portal as a repository, except for Gradle plugin projects.</p>
<p>Please note that Bintray also allowed any user to run their own public repository with a custom URL. This is also affected by the shutdown. You should check if you are using a repository with a URL starting with dl.bintray.com. Since these repositories can contain anything, you’ll need to figure out which dependencies you use from these repositories and where they may be migrating to.</p>
<h4 id="check-your-build-scan">Check your build scan</h4>
<p>If you’re using <a href="https://gradle.com/">Gradle Enterprise</a> or the public <a href="https://scans.gradle.com">build scans</a> service, you can check which repository was <a href="https://gradle.com/enterprise/releases/2018.4/#inspect-dependency-repositories-used-to-obtain-dependencies">used to resolve your dependencies</a>:</p>
<p><img src="/images/jcenter-shutdown/repositories-build-scan.png" alt="Repositories in a build scan" /></p>
<p>This will help you identify which dependencies are most likely to be affected by the shutdown.</p>
<h4 id="check-your-build-files">Check your build files</h4>
<p>If you declare your repositories in your build scripts, you can look for <code class="language-plaintext highlighter-rouge">jcenter()</code> or the URL jcenter.bintray.com.</p>
<p>You’re looking for something like</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>repositories {
jcenter()
}
</code></pre></div></div>
<p>or</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>repositories {
maven {
url = "https://jcenter.bintray.com"
}
}
</code></pre></div></div>
<p>or</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>repositories {
maven {
url = "https://dl.bintray.com/<some user name>"
}
}
</code></pre></div></div>
<h4 id="check-your-plugins">Check your plugins</h4>
<p>Any custom plugins could add a repository to your project. Built-in Gradle plugins (like <code class="language-plaintext highlighter-rouge">java-library</code>) do not add repositories to your project.</p>
<p>You’re looking for plugins that add JCenter as a repository programmatically with the <code class="language-plaintext highlighter-rouge">jcenter()</code> API.</p>
<h4 id="check-with-your-team">Check with your team</h4>
<p>You may not find any references to JCenter in your build scripts or plugins, but you may still be using JCenter. If you have an internal mirror or corporate proxy repository that you use, you need to check if that repository uses JCenter.</p>
<p>The impact to paid JFrog Cloud customers may be different, so please contact JFrog directly to understand if you can continue to use JCenter if your internal mirror is provided by JFrog.</p>
<p><a name="troubleshooting"></a></p>
<h3 id="troubleshooting-dependencies-from-jcenter">Troubleshooting dependencies from JCenter</h3>
<p>You can use <code class="language-plaintext highlighter-rouge">gradle dependencies</code> (or build scans) to determine which dependencies were not resolved or how they may have changed. The Gradle build will also fail when resolution errors if it’s unable to resolve a dependency from any repository.</p>
<p>In the example below, you can see that Gradle failed to resolve the trove4j dependency, after switching to Maven Central.</p>
<p>On the console:
<img src="/images/jcenter-shutdown/unresolved-console.png" alt="Unresolved dependencies on the console" /></p>
<p>In a build scan:
<img src="/images/jcenter-shutdown/unresolved-build-scan.png" alt="Unresolved dependencies in a build scan" /></p>
<p>At this point, if the package is only available on JCenter, your options are to:</p>
<ul>
<li>Wait for the maintainer to migrate to another repository (e.g., Maven Central)</li>
<li>Find another equivalent package that is available on Maven Central</li>
<li>Remove your dependency on the package entirely</li>
<li>Copy the package to your own internal repository</li>
</ul>
<p>If your build can resolve most of its dependencies from another repository, you could continue to use JCenter for the few packages that are only available there. You can prevent new packages that are only available on JCenter from being introduced into your build, by using a <a href="https://docs.gradle.org/current/userguide/declaring_repositories.html#sec:repository-content-filtering">content filter</a>. This prevents Gradle from looking in JCenter for all packages.</p>
<p>For instance, many Android projects depend on the Trove4J library that at the time of writing is only available from JCenter. We can use a <a href="https://github.com/gradle/android-cache-fix-gradle-plugin/blob/ed384a0f796627a6492d0811eccb90629b575368/build.gradle#L27-L32">content filter to only allow artifacts in the <code class="language-plaintext highlighter-rouge">org.jetbrains.trove4j</code> group</a> to come from JCenter. We also put <code class="language-plaintext highlighter-rouge">jcenter()</code> last so that Maven Central is searched first.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>repositories {
mavenCentral()
jcenter {
content {
// org.jetbrains.trove4j is only available in JCenter
includeGroup("org.jetbrains.trove4j")
}
}
}
</code></pre></div></div>
<p><a name="impact-plugins"></a></p>
<h2 id="impact-to-gradle-plugins">Impact to Gradle plugins</h2>
<p>Behind the scenes, the <a href="https://plugins.gradle.org/">Gradle Plugin Portal</a> uses JCenter to resolve dependencies of plugins. We will be migrating the Plugin Portal away from JCenter before the final shutdown. Builds will not need to make changes while the Plugin Portal migrates away from JCenter.</p>
<h3 id="resolving-existing-plugins">Resolving existing plugins</h3>
<p>Existing plugins will continue to resolve in the same way they do today. No changes should need to be made to your builds.</p>
<h3 id="publishing-new-or-updated-plugins">Publishing new or updated plugins</h3>
<p>If you are publishing your plugin with the <code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code>, you are not affected. Just double check that your build does not use JCenter directly.</p>
<p>If you are publishing your plugin to Bintray, you will need to use the <a href="https://plugins.gradle.org/docs/bintray"><code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code> plugin</a> to publish new versions of your plugin. The sooner you can do this, the better. We can help if you have questions or problems.</p>
<p>We will automatically migrate all plugins published to Bintray on March 31, 2021. After that date, we will no longer synchronize plugins published to Bintray. This migration should be transparent to users resolving dependencies.</p>
<p><a name="impact-publishing"></a></p>
<h2 id="impact-to-packages-publishing-to-bintray">Impact to packages publishing to Bintray</h2>
<p>If you are publishing your builds to Bintray using the <a href="https://github.com/bintray/gradle-bintray-plugin">com.jfrog.bintray</a> plugin, you’ll need to find another repository to host your package. You may be able to publish using the <a href="https://docs.gradle.org/current/userguide/publishing_setup.html#header">built-in <code class="language-plaintext highlighter-rouge">maven-publish</code></a>.</p>
<p>If you’re migrating to Maven Central, the <a href="https://github.com/gradle-nexus/publish-plugin">io.github.gradle-nexus.publish-plugin</a> plugin automates publishing to Nexus and Maven Central.</p>
<p><a name="security-considerations"></a></p>
<h2 id="security-considerations">Security considerations</h2>
<p>There are a few security considerations you should make before migrating from JCenter to another repository.</p>
<p>As we have seen in recent years, <a href="https://blog.autsoft.hu/a-confusing-dependency/">dependency confusion</a> and <a href="https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610">namespace collisions</a> can be serious dangers for supply chain attacks. Each of these vulnerabilities relied on untrusted artifacts being used in place of trusted ones.</p>
<p>When you change the list of repositories you use to resolve dependencies, you may inadvertently expose yourself to these kinds of attacks.</p>
<p>For instance, assume you have a build that uses JCenter and another less popular repository (a.k.a, LessPopularCenter) to resolve dependencies (in that order) and your build has a dependency on <code class="language-plaintext highlighter-rouge">com.example:foo:1.0</code>. Package <code class="language-plaintext highlighter-rouge">com.example:foo</code> was only published to JCenter, but someone else has managed to put a copy of <code class="language-plaintext highlighter-rouge">com.example:foo:1.0</code> on LessPopularCenter. As long as JCenter is the first place Gradle checks for dependencies, your build would always get the official package.</p>
<p>If you remove JCenter and replace it with Maven Central (which doesn’t have a copy of <code class="language-plaintext highlighter-rouge">com.example:foo</code>), your build may still be able to resolve <code class="language-plaintext highlighter-rouge">com.example:foo</code> because LessPopularCenter has a copy of it. Unfortunately, there are no guarantees that the copy on LessPopularCenter is the same as the official package.</p>
<p>If you want a strong assurance that the artifacts that you rely on now are the same or come from the same trusted party, you should enable <a href="https://docs.gradle.org/current/userguide/dependency_verification.html#sub:enabling-verification">dependency verification</a>. Once enabled, Gradle automatically performs dependency verification for all dependencies resolved from repositories. This will help you detect unexpected changes to a dependency’s artifacts, compromised artifacts or new untrusted sources of artifacts.</p>
<p>Dependency verification requires some up-front setup. You need to specify a list of trusted keys to use when verifying signed artifacts. For unsigned artifacts, you need to specify expected checksums for each artifact.</p>
<p>Dependency verification failures don’t necessarily mean that something nefarious has happened. Dependency metadata and artifacts are messy. There are sometimes legitimate reasons for a package to have different artifacts from two repositories for the same version. If the package was published to JCenter and Maven Central separately, it could have been built twice and produced different artifacts that behave the same. This means that even when you resolve the “same” version from both repositories, when you compare the artifacts byte-for-byte, you will get different results. In that case, you could tell Gradle to <a href="https://docs.gradle.org/current/userguide/dependency_verification.html#sec:trusting-several-checksums">accept both checksums as valid</a>.</p>
<h2 id="feedback">Feedback</h2>
<p>Let us know if you have any questions on our <a href="https://discuss.gradle.org/">forums</a> or <a href="https://gradle.com/slack-invite">Gradle Community Slack</a>.</p>
Introducing the Gradle Fellowship Program2021-02-10T00:00:00-05:00https://blog.gradle.org/gradle-fellowshipPiotr Jagielski
<p>Community is instrumental to the success of every open-source project. At Gradle, we are very fortunate to serve and collaborate with an amazing community of users, champions, and plugin authors.</p>
<p>Some of the community members go above and beyond to help other users succeed. We are extremely grateful for their dedication. Their valuable contributions to the community include helping others on the <a href="https://discuss.gradle.org/">Gradle Forums</a> and <a href="https://gradle.com/slack-invite">Community Slack</a>, publishing blog posts and presenting at conferences and user groups.</p>
<p>We are excited to introduce the Gradle Fellowship program to recognize and connect those individuals and grow this global community of Gradle experts to help the broader Gradle community be more productive.</p>
<p>We’ll support the Gradle Fellows in their efforts by offering additional direct communication channels with Gradle developers. Through these interactions, we will strive to make the latest developments in Gradle more transparent to the community and gain a better understanding of the community’s needs.</p>
<p>You can find the list of Gradle Fellows on our website at <a href="https://gradle.org/fellows">gradle.org/fellows</a>. We are starting with a small group of some of the most active and long-standing community champions. We are looking forward to expanding this group in the upcoming months.</p>
Introducing Java toolchains2020-10-30T00:00:00-04:00https://blog.gradle.org/java-toolchainsLouis Jacomet
<p>Building a Java project and running Gradle both require the installation of a JDK.
Most Gradle users conveniently use the Java installation that is running Gradle to build and test their application.</p>
<p>While this is acceptable in simple cases, there are a number of issues with this approach. For example, if a build requires the same Java version to be used on all machines, each developer needs to know about that requirement and install that version manually.</p>
<p>It has been possible to configure Gradle to build a project with a different Java version than the one used to run Gradle. However, it has required configuring each task like compilation, test, and javadoc separately.</p>
<p>Gradle 6.7 introduces “Java toolchain support”.
In a nutshell, it allows you to run Gradle with whatever version of Java you have installed, while Gradle builds the project with the version of Java declared in a single place. Gradle will automatically configure the build, detect already installed JDKs and download the required JDK if it is not already installed.</p>
<p>No more <code class="language-plaintext highlighter-rouge">release</code> or <code class="language-plaintext highlighter-rouge">sourceCompatibility</code> tweaks, no more wiki pages describing which JDK you should install for the build to work. The advantages are tremendous, let’s see how to use this!</p>
<h2 id="introducing-java-toolchains-support">Introducing Java toolchains support</h2>
<p>Java toolchains support enables build authors to declare which Java version their project requires for compiling, testing and running their code.</p>
<p style="text-align:center;"><iframe width="560" height="315" src="https://www.youtube.com/embed/ANAljTJcuqM" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe></p>
<h3 id="project-wide-configuration">Project wide configuration</h3>
<p>Let’s start with what you, as a build author, need to do, at the minimum:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>plugins {
id("java-library") // or id("application")
}
java {
toolchain {
languageVersion.set(JavaLanguageVersion.of(11))
}
}
</code></pre></div></div>
<p>And that’s it!</p>
<p>What do you get with the above in Gradle 6.7?</p>
<p>You get:</p>
<ul>
<li>All Java compilation tasks will use Java 11 to build.
That means code in the <code class="language-plaintext highlighter-rouge">main</code> and <code class="language-plaintext highlighter-rouge">test</code> source sets, but also Java code in any custom source set you add, will be built with the configured Java version.</li>
<li>All tests tasks, including the default <code class="language-plaintext highlighter-rouge">test</code> task and any additional custom <code class="language-plaintext highlighter-rouge">Test</code> task, will use Java 11 to run the tests.</li>
<li>The <code class="language-plaintext highlighter-rouge">javadoc</code> task will use Java 11 to build the documentation.</li>
</ul>
<p>In addition, with the <code class="language-plaintext highlighter-rouge">application</code> plugin, the <code class="language-plaintext highlighter-rouge">run</code> task will use Java 11 to start your application.</p>
<p>Behind the scenes, Gradle will have:</p>
<ul>
<li>Located a Java 11 installation, from a list of <a href="https://docs.gradle.org/6.7/userguide/toolchains.html#sec:auto_detection">known locations</a> such as <a href="https://github.com/halcyon/asdf-java">asdf</a>, <a href="https://github.com/shyiko/jabba">jabba</a>, <a href="https://sdkman.io/">SDKMAN!</a> or standard locations per operating system</li>
<li>Validated the installation and its version</li>
<li>If no Java 11 installation was found, Gradle will attempt to <a href="https://docs.gradle.org/6.7/userguide/toolchains.html#sec:provisioning">download one</a> from <a href="https://adoptopenjdk.net/">AdoptOpenJDK</a></li>
</ul>
<p>If Gradle cannot find the requested Java version, you can tell it <a href="https://docs.gradle.org/6.7/userguide/toolchains.html#sec:custom_loc">where to look</a>.</p>
<h3 id="task-level-configuration">Task level configuration</h3>
<p>In some cases, you need to use a different toolchain for a specific task.
Let’s say the library you develop has a specific code path depending on the Java version it runs on.
You will want to make sure both code paths are exercised and so will run a set of tests with a different Java version.
This scenario is supported as follows.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tasks.register<Test>("extraTests") {
javaLauncher.set(javaToolchains.launcherFor {
languageVersion.set(JavaLanguageVersion.of(14))
})
}
</code></pre></div></div>
<p>Head over to <a href="https://docs.gradle.org/6.7-rc-1/userguide/toolchains.html#specify_custom_toolchains_for_individual_tasks">the documentation</a> to learn more about it.</p>
<h2 id="further-benefits-of-toolchain-usage">Further benefits of toolchain usage</h2>
<p>In addition to the benefits we have discussed earlier, toolchains can help in more situations.</p>
<p>It was problematic for Gradle to build code on not yet released Java versions – like Java 16 at the time of writing – which may fail to run existing releases of Gradle – such as the current 6.7 release.</p>
<p>With toolchains support, Gradle 6.7 is perfectly capable of compiling and testing code with a not yet released language version.
You only need to ask it to do it:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java {
toolchain {
languageVersion.set(JavaLanguageVersion.of(16))
}
}
</code></pre></div></div>
<p>Additionally, the code may behave differently when built and run with different Java versions, potentially causing hard to diagnose issues.</p>
<p>Even without incompatibilities affecting the system behavior, using different Java versions to build the project on different machines can reduce build performance because of <a href="https://docs.gradle.org/6.7/userguide/build_cache.html">build cache</a> misses.</p>
<h3 id="relaxing-java-version-constraints-on-gradle">Relaxing Java version constraints on Gradle</h3>
<p>In addition to the benefits for build authors, this feature will have an impact on the development of Gradle itself.
Up to now Gradle assumed that it would run with the same version of Java as required by your project.
This means that Gradle has to be able to run with a set of Java versions spanning the needs of the projects using Gradle.
With the release cadence of Java 5 to 8 this was probably acceptable.
With the new Java release schedule, this is more problematic.</p>
<p>As of version 6.7, Gradle requires:</p>
<ul>
<li>Java 8 at the minimum to run a build,</li>
<li>is built daily with 11</li>
<li>and tested up to Java 15.</li>
</ul>
<p>All of that requires quite a sophisticated CI setup and imposes constraints on Gradle core development.
Features and APIs beyond Java 8 are not accessible to Gradle engineers for developing the tool for example.</p>
<p>Once toolchains are adopted in the community, we will eventually be able to be less conservative with Gradle’s runtime requirements. Gradle may then only run on relatively recent Java versions while supporting to build code for older Java versions through toolchains.</p>
<h2 id="whats-next-for-toolchains">What’s next for toolchains?</h2>
<p>In Gradle 6.7, the toolchain support is limited to the tasks known by the Java dedicated plugins:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">JavaCompile</code></li>
<li><code class="language-plaintext highlighter-rouge">Test</code></li>
<li><code class="language-plaintext highlighter-rouge">Javadoc</code></li>
<li><code class="language-plaintext highlighter-rouge">JavaExec</code></li>
</ul>
<p>Toolchain support for the Groovy and Scala plugins is planned to be added in one of the next Gradle releases.</p>
<p>We also hope that given the benefits of this feature for users, community <a href="https://docs.gradle.org/6.7/userguide/toolchains.html#sec:plugins">plugin authors will leverage the toolchain support</a> in their own tasks.</p>
Introducing Configuration Caching2020-08-10T00:00:00-04:00https://blog.gradle.org/introducing-configuration-cachingPaul Merlin
<p><em>This is the second installment in a series of blog posts about incremental development — the part of the software development process where you make frequent small changes. We will be discussing upcoming Gradle build tool features that significantly improve feedback time around this use case. In the previous post, we <a href="https://blog.gradle.org/introducing-file-system-watching">introduced file system watching</a> for Gradle 6.5.</em></p>
<p>In <a href="https://docs.gradle.org/6.6/release-notes.html">Gradle 6.6</a> we are introducing an experimental feature called the configuration cache that significantly improves build performance by caching the result of the <a href="https://docs.gradle.org/current/userguide/build_lifecycle.html#build_lifecycle">configuration phase</a> and reusing this for subsequent builds. Using the configuration cache, Gradle can skip the configuration phase entirely when nothing that affects the build configuration, such as build scripts, has changed.</p>
<link href="https://cdnjs.cloudflare.com/ajax/libs/c3/0.7.15/c3.min.css" rel="stylesheet" type="text/css" />
<style>
.c3 text { font-size: 1.5em; }
.c3-legend-item text { font-size: 1.2em; }
</style>
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.16.0/d3.min.js" charset="utf-8"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/c3/0.7.15/c3.min.js"></script>
<script type="text/javascript">
function chartPerformance(elementSelector, title, data) {
c3.generate({
bindto: elementSelector,
data: {
rows: data,
type: 'bar',
x: 'Scenario',
labels: true
},
axis: {
x: { type: 'category' },
y: { label: 'duration (s)' }
},
legend: { position: 'bottom' },
title: { text: title },
color: { pattern: ['#1BA8CB', '#00CB9C', '#898989', '#02303A'] },
tooltip: { format: { value: function (value, ratio, id, index) { return value + 's'; } } }
});
}
function chartPerformanceStacked(elementSelector, title, data, stackedGroups, categories) {
c3.generate({
bindto: elementSelector,
data: {
columns: data,
type: 'bar',
groups: stackedGroups,
labels: true
},
axis: {
x: {
type: 'category',
categories: categories
},
y: { label: 'duration (ms)' }
},
legend: { position: 'bottom' },
title: { text: title },
color: { pattern: ['#1BA8CB', '#00CB9C', '#898989', '#02303A'] },
tooltip: { format: { value: function (value, ratio, id, index) { return value + 'ms'; } } }
});
}
</script>
<p>On top of that, when reusing the configuration cache, more work is run in parallel by default and dependency resolution is cached. The isolation of the configuration and execution phases, and the isolation of tasks, make these optimizations possible.</p>
<p>Note that configuration caching is different from the <a href="https://docs.gradle.org/current/userguide/build_cache.html">build cache</a>, which caches outputs produced by the build. The configuration cache captures only the state of the configuration phase. It’s also separate from IDE sync and import processes that do not currently benefit from configuration caching.</p>
<p>In order to cache the configuration result, Gradle applies some strong <a href="https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:requirements">requirements</a> that plugins and build scripts need to follow. Many plugins, including some core Gradle plugins, do not meet these requirements yet. Moreover, support for configuration cache in some Gradle features is not yet implemented. Therefore, your build and the plugins you depend on will likely require changes to fulfil the requirements. Gradle will <a href="https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:troubleshooting">report problems</a> found with your build logic to assist you in making your build work with the configuration cache.</p>
<p>The configuration cache is currently <strong><em>highly experimental</em></strong> and not enabled by default. We release it early in order to collect feedback from the community while we work on stabilizing the new feature.</p>
<p>That being said, we are committed to making the configuration cache production-ready with the ultimate goal to enable it by default. You can expect that it will get significantly better in the next Gradle releases.</p>
<h2 id="configuration-caching-in-action">Configuration caching in action</h2>
<p>It is recommended to get started with the simplest task invocation possible. Running <code class="language-plaintext highlighter-rouge">help</code> with the configuration cache enabled is a good first step:</p>
<p><img src="images/introducing-configuration-caching/running-help.gif" alt="Running help" /></p>
<p>Here it is in action from Android Studio, deploying the middle sized <a href="https://github.com/gradle/santa-tracker-performance">Santa Tracker Android</a> application to an Android emulator, making changes to how the snowflakes move on the screen and applying the changes to the emulator:</p>
<video autoplay="" controls="" width="699" height="466">
<source src="/images/introducing-configuration-caching/deploy-santa-tracker.mp4" type="video/mp4" />
<source src="/images/introducing-configuration-caching/deploy-santa-tracker.webm" type="video/webm" />
</video>
<h2 id="build-time-improvements">Build time improvements</h2>
<p>The practical impact of the feature depends on a number of factors, but in general it should result in a significant reduction of build time. We’ve seen drastic improvements on large real world builds, let’s have a look at some of them.</p>
<h3 id="java-builds">Java builds</h3>
<p>On a large Java enterprise build with ~500 subprojects and complex build logic, running <code class="language-plaintext highlighter-rouge">:help</code> went from 8 seconds down to 0.5 seconds. That’s 16 times faster. Of course, running <code class="language-plaintext highlighter-rouge">:help</code> isn’t that useful but it gives an idea of the saved time for the configuration phase. On the same build, running <code class="language-plaintext highlighter-rouge">assemble</code> after changing some implementation code went from ~40 seconds down to ~13 seconds, that’s ~3 times faster.</p>
<div id="chart-large-java"></div>
<script>
chartPerformance(
'#chart-large-java',
'Large Java Enterprise - Build time in seconds',
[
['Scenario', 'Baseline', 'Configuration Cache'],
[':help', 8, 0.5],
[':assemble (non-abi change)', 40, 13]
]
);
</script>
<p>Now let’s look at the <a href="https://github.com/gradle/gradle">gradle/gradle</a> build. It has a hundred subprojects and a fairly complex build logic. You can use it to reproduce these results. Running a test after making an implementation change goes from 16.4 seconds down to 13.8 seconds, skipping the ~2 seconds configuration phase:</p>
<div id="chart-gradle-gradle"></div>
<script>
chartPerformanceStacked(
'#chart-gradle-gradle',
'gradle/gradle - Build time in milliseconds',
[
['configuration phase', 2211, 214],
['execution phase', 16390-2211, 13887-214]
],
[['configuration phase', 'execution phase']],
['Baseline', 'Configuration Cache']
);
</script>
<p>In blue you can see the configuration phase, in green the execution phase. On the left, without the configuration cache enabled, configuration phase takes more than 2 seconds and goes down to 214 milliseconds with the configuration cache on the right.</p>
<p>You can also see that the execution phase benefits from the configuration cache but is dominated by compiling and running the tests in that case.</p>
<h3 id="android-builds">Android builds</h3>
<p>Another notable example is a very large real world Android build with ~2500 subprojects. On that build, running <code class="language-plaintext highlighter-rouge">:help</code> went from ~25 seconds down to ~0.5 seconds, that’s 50 times faster! Running a more useful build such as assembling the APK after changing some implementation, goes from ~50 seconds down to ~20 seconds, almost 3 times faster.</p>
<div id="chart-large-android"></div>
<script>
chartPerformance(
'#chart-large-android',
'Large Android App - Build time in seconds',
[
['Scenario', 'Baseline', 'Configuration Cache'],
[':help', 25, 0.5],
[':assemble (non-abi change)', 50, 20]
]
);
</script>
<p>In the <a href="https://github.com/gradle/santa-tracker-performance">Santa Tracker Android</a> project, we’ve seen the following improvements in the build time for a small implementation change:</p>
<div id="chart-santa-tracker"></div>
<script>
chartPerformanceStacked(
'#chart-santa-tracker',
'Santa Tracker App - Build time in milliseconds',
[
['configuration phase', 129, 63.5],
['execution phase', 897.5-129, 656-63.5]
],
[['configuration phase', 'execution phase']],
['Baseline', 'Configuration Cache']
);
</script>
<p>The configuration phase is cut in half, from 129 milliseconds down to 63.5 milliseconds. You can also see that the execution phase is accelerated by the configuration cache due to more task parallelisation and caching of dependency resolution.</p>
<p>If you want to reproduce with the above builds or measure your own builds you can use the <a href="https://github.com/gradle/gradle-profiler">Gradle Profiler</a> by following the instructions in <a href="https://github.com/gradle/gradle-benchmark-base">this repository</a>. Note that the Gradle Profiler will show a slightly different picture, closer to the experience from IDEs, because both use the Gradle Tooling API. This skips the fixed cost of starting the Gradle client JVM that happens when you use the command line.</p>
<h2 id="how-does-it-work">How does it work?</h2>
<p>When the configuration cache is enabled and you run Gradle for a particular set of tasks, for example by running <code class="language-plaintext highlighter-rouge">./gradlew check</code>, Gradle checks whether a configuration cache entry is available for the requested set of tasks. If available, Gradle uses this entry instead of running the configuration phase. The cache entry contains information about the set of tasks to run, along with their configuration and dependency information.</p>
<p>The first time you run a particular set of tasks, there will be no entry in the configuration cache for these tasks and so Gradle will run the configuration phase as normal:</p>
<ol>
<li>Run init scripts.</li>
<li>Run the settings script for the build, applying any requested settings plugins.</li>
<li>Configure and build the buildSrc project, if present.</li>
<li>Run the builds scripts for the build, applying any requested project plugins.</li>
<li>Calculate the task graph for the requested tasks, running any deferred configuration actions.</li>
</ol>
<p>Following the configuration phase, Gradle writes the state of the task graph to the configuration cache, taking a snapshot for later Gradle invocations. The execution phase then runs as normal. This means you will not see any build performance improvement the first time you run a particular set of tasks.</p>
<p>When you subsequently run Gradle with this same set of tasks, for example by running <code class="language-plaintext highlighter-rouge">./gradlew check</code> again, Gradle will load the tasks and their configuration directly from the configuration cache, skip the configuration phase entirely and run all tasks in parallel. Before using a configuration cache entry, Gradle checks that none of the “build configuration inputs”, such as build scripts, for the entry has changed. If a build configuration input has changed, Gradle will not use the entry and will run the configuration phase again as above, saving the result for later reuse.</p>
<h2 id="requirements-and-limitations">Requirements and limitations</h2>
<p>In order to capture the state of the task graph into the configuration cache and reload it again in a later build, Gradle applies certain requirements to tasks and other build logic. Each of these requirements is treated as a configuration cache “problem” and by default causes the build to fail if violations are present.</p>
<p>If any problem is found caching or reusing the configuration, an HTML report is generated to help you diagnose and fix the issues.</p>
<p><img src="images/introducing-configuration-caching/problems-report.png" alt="Problems Report" /></p>
<p>If you encounter such problems, your build or the Gradle plugins in use probably need to be adjusted. See the <a href="https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:troubleshooting">Troubleshooting</a> section of the documentation for more information about how to use this report.</p>
<p>You can find the set of <a href="https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:plugins">supported core plugins</a> and the set of <a href="https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:not_yet_implemented">not yet implemented Gradle features</a> in the configuration cache documentation.</p>
<p>The latest Android Gradle Plugin preview, <code class="language-plaintext highlighter-rouge">4.2.0-alpha07</code> at the time of writing, works with the configuration cache.
The latest Kotlin Gradle Plugin, <code class="language-plaintext highlighter-rouge">1.4.0-RC</code> at the time of writing, works on simple JVM projects emitting some problems.
Kotlin <code class="language-plaintext highlighter-rouge">1.4.20</code> is the current target for a fully compliant plugin.
This information can be found at <a href="https://github.com/gradle/gradle/issues/13490">gradle/gradle#13490</a> alongside the status of the most used community plugins.</p>
<h2 id="try-out-configuration-caching">Try out configuration caching</h2>
<p>If you would like to see how your project benefits from configuration caching, here is how you can try it out.</p>
<p>First, make sure you run <a href="https://docs.gradle.org/6.6/release-notes.html">Gradle 6.6</a> or later. In order to <a href="https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:usage">enable configuration caching</a>, you need to pass <code class="language-plaintext highlighter-rouge">--configuration-cache</code> on the command line. Alternatively, add</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>org.gradle.unsafe.configuration-cache=true
</code></pre></div></div>
<p>to the <code class="language-plaintext highlighter-rouge">gradle.properties</code> file in the project directory or in the Gradle user home, so you don’t need to pass the command-line option on every build. That’s it: the next build will run with configuration caching enabled.</p>
<p>Keep in mind that you will only see performance improvements when subsequent builds with the same requested tasks have the feature enabled. If you want to benchmark your build, you can do it easily with <a href="https://github.com/gradle/gradle-profiler">Gradle Profiler</a> by following the instructions in <a href="https://github.com/gradle/gradle-benchmark-base">this repository</a>.</p>
<p>If you run into any problems, check out the supported <a href="https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:plugins">core plugins</a> or <a href="https://github.com/gradle/gradle/issues/13490">community plugins</a>, learn how to <a href="https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:troubleshooting">troubleshoot</a> in the user manual. You can also read our recommended <a href="https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:adoption">adoption steps</a>.</p>
<p>If you still have problems <a href="https://github.com/gradle/gradle/issues">open a Gradle issue</a> if you think the problem is with Gradle, or check the supported community plugins issue at <a href="https://github.com/gradle/gradle/issues/13490">gradle/gradle#13490</a>. You can also get help in the <code class="language-plaintext highlighter-rouge">#configuration-cache</code> channel in the <a href="https://gradle-community.slack.com/archives/C013WEPGQF9">Gradle community Slack</a>.</p>
Gradle vs Bazel for JVM Projects2020-06-30T00:00:00-04:00https://blog.gradle.org/gradle-vs-bazel-jvmPiotr Jagielski
<link href="https://cdnjs.cloudflare.com/ajax/libs/c3/0.7.15/c3.min.css" rel="stylesheet" type="text/css" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.16.0/d3.min.js" charset="utf-8"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/c3/0.7.15/c3.min.js"></script>
<script type="text/javascript">
// Usage: chartPerformance('#commons-lang', 'Title', [['Scenario', 'Gradle 5.4', 'Maven 3.6'], ['My Scenario', 21.5, 38.3]]])
function chartPerformance(elementSelector, title, data) {
c3.generate({
bindto: elementSelector,
data: {
rows: data,
type: 'bar',
x: 'Scenario',
labels: true
},
axis: {
x: {
type: 'category'
},
y: {
label: 'duration (s)'
}
},
legend: {
position: 'bottom'
},
title: {
text: title
},
color: {
pattern: ['#1BA8CB', '#00CB9C', '#898989', '#02303A']
},
tooltip: {
format: {
value: function (value, ratio, id, index) { return value + 's'; }
}
}
});
}
</script>
<h2 id="introduction">Introduction</h2>
<p>Gradle has emerged as the build tool of choice for projects within the JVM ecosystem, including Kotlin. It is the most popular build tool for open source JVM projects on GitHub. It is downloaded on average more than 15 million times per month and has been counted in the <a href="https://techcrunch.com/2017/04/07/tracking-the-explosive-growth-of-open-source-software/">Top 20 Most Popular Open Source Projects for IT</a> by Techcrunch. Many popular projects have migrated from Maven to Gradle, with <a href="https://spring.io/blog/2020/06/08/migrating-spring-boot-s-build-to-gradle">Spring Boot</a> being a prominent example.</p>
<p>Recently, we have received inquiries about the suitability of Google’s Bazel build tool for usage within JVM environments. What follows is a detailed comparative performance analysis and evaluation of key capabilities, that arrives at three major conclusions:</p>
<ul>
<li>Despite Bazel’s strong and well-deserved reputation for performance and scalability, Gradle outperforms Bazel in almost every scenario we tested.</li>
<li>Optimizing projects for Bazel comes at a significant cost for build authoring and maintenance. </li>
<li>Gradle provides more compelling features and conveniences for common use cases in JVM projects.</li>
</ul>
<p>In summary, the data and analysis indicates clearly that Gradle is a better choice than Bazel for most JVM projects. We will provide an equivalent comparison for Android projects in a follow up article.</p>
<p>At the same time, recognizing that individual tools have unique strengths in addressing the needs and requirements of specific developer ecosystems and specific use cases, we expect that build tool specialization and fragmentation will continue to be the norm across and even within these ecosystems (e.g. Gradle and Maven).</p>
<p>In recognition of this, <a href="https://gradle.com/">Gradle Enterprise</a> provides analytics and acceleration for more than just Gradle. This allows users to benefit from faster and more reliable builds without migrating to any specific build tool. Today this includes Gradle and Maven, but will include Bazel and other tools in the future. To learn more about the acceleration technology supported in Gradle Enterprise for the Gradle, Maven, and soon Bazel build tools, watch this video entitled “<a href="https://tv.gradle.com/maven-build-cache-demo">Speed Up Maven Builds | Maven Build Cache Technology & Business Case Explained</a>.”</p>
<h2 id="performance">Performance</h2>
<p>How well a build system performs for your project depends on a variety of factors. These include the size and structure of your project, the toolchain you are using, and your workflows. In this section we describe our approach to making performance comparisons and the results based on representative scenarios.</p>
<h3 id="about-the-performance-comparison">About the Performance Comparison</h3>
<p>Gradle and Bazel have different approaches and areas of focus with respect to performance optimization. It would be easy to create some test projects to show that either Gradle is dramatically faster than Bazel or vice versa for a particular performance scenario. The question is always how applicable the results of such test projects are for real life projects and, more importantly, for your project.</p>
<p>Our approach for this comparison is to model common Java project types. This ranges from large repositories with many million lines of code to small library/microservices projects. In this article we will not cover the advantages and disadvantages of the various ways to approach structuring code, either within a single large source repository or across many smaller source repositories. Both approaches have a significant footprint in the JVM ecosystem and often also within a single organization. It requires a serious migration effort and change of workflows and culture to switch from one approach to another. A build system for the JVM should support both approaches well.</p>
<h3 id="performance-measurement-scenarios">Performance Measurement Scenarios</h3>
<p>The different scenarios are measured with the following project sizes and shapes:</p>
<table>
<thead>
<tr>
<th>Project name</th>
<th>No of modules</th>
<th>No of production classes</th>
<th>No of test classes</th>
<th>LOC</th>
</tr>
</thead>
<tbody>
<tr>
<td>Small multi-project</td>
<td>10</td>
<td>500</td>
<td>500</td>
<td>90k</td>
</tr>
<tr>
<td>Medium multi-project</td>
<td>100</td>
<td>10 000</td>
<td>10000</td>
<td>1.8M</td>
</tr>
<tr>
<td>Large multi-project</td>
<td>500</td>
<td>50000</td>
<td>50000</td>
<td>9M</td>
</tr>
<tr>
<td>Large monolithic</td>
<td>1</td>
<td>50000</td>
<td>50000</td>
<td>7M</td>
</tr>
</tbody>
</table>
<p>All of the test projects are open source, including the runners that were used to take the measurements. See the <a href="https://github.com/gradle/bazel-comparison">instructions</a> for details on how to reproduce the measurements.</p>
<blockquote>
<p>For each scenario below, Gradle is benchmarked using the latest performance optimizations of Gradle 6.5 including a <a href="https://docs.gradle.org/current/userguide/configuration_cache.html">configuration cache</a> and <a href="https://docs.gradle.org/current/userguide/gradle_daemon.html#sec:daemon_watch_fs">file-system watching</a>. Some of the optimizations used are experimental and were enabled by feature flags.</p>
</blockquote>
<h3 id="scenario-1-full-build">Scenario 1: Full Build</h3>
<p>This scenario is a full build with no existing output in the build output directory and no local or remote build cache available. The external dependencies are available in the local dependency cache. This build includes building all class files of the project, packaging them as JAR as well as running all unit tests.</p>
<p>Both Gradle and Bazel users rarely run full builds locally. Even on CI, completely full builds can usually be avoided thanks to build caching (more on that later). Still, this is an interesting scenario because it approximates a change that affects a large portion of the codebase, like an API change in a library that is used by most of the codebase.</p>
<p>Additionally this scenario is interesting for Maven users to see if the performance advantages over Maven come from caching or other factors since they usually do clean builds for reliability reasons.</p>
<div id="full-build"></div>
<script>
var data = [
['Scenario', 'Gradle', 'Bazel', 'Maven'],
['Small', 6.2, 14, 17.03],
['Medium', 49.4, 89, 177.86],
['Large', 180, 307, 688.23],
['Monolith', 215.6, 215.5, 289.90]];
chartPerformance('#full-build', 'Build time in seconds', data);
</script>
<p>Gradle and Bazel both have performance advantages over Maven even without build caching and optimizations for incremental builds. Gradle is the winner in this category.</p>
<h3 id="scenario-2-remote-caching">Scenario 2: Remote Caching</h3>
<p>The most effective way to make builds and tests faster is by rebuilding only what has changed. There are multiple layers to this approach. One is a remote build cache. A remote build cache provides output across machines for build actions that have unchanged inputs.</p>
<p>There are two major workflows that get accelerated this way.</p>
<p>The first is the speed of your CI builds. CI builds run on many agents, often throwing away any previous state before they start. So they often build everything from scratch although the actual new commits they are building only require rebuilding a few areas of the codebase. With a remote build cache, build actions with inputs that are not affected by the new commits can retrieve the output from another, previous build on any other CI machine in your organization. This both dramatically reduces CI build time and saves significant compute resources.</p>
<p>A remote cache also helps speed up local builds whenever you pull the latest changes from version control and then run the build locally after a CI build has already populated the cache.</p>
<p>Both Gradle and Bazel provide support for remote caching. Managed cache backends are available for both tools (e.g. Gradle Enterprise and Google Cloud Storage). Gradle Enterprise also provides remote build caching for Maven.</p>
<p>This scenario executes a full build just like in scenario 1, but this time leverages a remote build cache that has matching output for every build action. This approximates an ephemeral (i.e. no re-use of previous local state) CI build agent running a build for changes that only affect a small part of the codebase or a first build on a local machine.</p>
<div id="remote-caching"></div>
<script>
var data = [
['Scenario', 'Gradle', 'Bazel', 'Maven'],
['Small', 2, 1.4, 5.2],
['Medium', 5.3, 12.4, 15.9],
['Large', 26.3, 62.4, 60.41]];
chartPerformance('#remote-caching', 'Build time in seconds', data);
</script>
<p>Gradle also provides the best performance in this scenario.</p>
<p>Note that the remote caching scenarios are the hardest to compare as users get significantly different results depending on their network connection speed and their proximity to the remote cache node. For all tests, the remote cache was deployed to a geographically close data center and accessed via a fast and reliable home Internet connection.</p>
<p>The comparison does not include the “monolith” scenario where all the code is in a single module, because remote caching offers little benefit for such projects.</p>
<h3 id="scenario-3-incremental-builds">Scenario 3: Incremental builds</h3>
<p>One of the most important aspects of developer productivity is the feedback time for local incremental builds. It can be measured by how long it takes for developer builds to get feedback after one or a few lines of code have been changed. Reusing local state from previous build runs on the same machine is key to perform well in those scenarios. As nothing needs to be sent over the wire, or very often not even copied, this is faster than using a remote cache.</p>
<p>In the next scenario, the project has been built successfully. Another build is run after a single line of code was changed. This is an approximation of a typical local build. There are two variations of this scenario—with and without a change to a public interface of a class/method. A change not affecting the API allows build tools to apply additional optimizations.</p>
<div id="incremental-changing-api"></div>
<script>
var data = [
['Scenario', 'Gradle', 'Bazel', 'Maven (no clean)', 'Maven (local cache)'],
['Small', 1.01, 1.48, 7.33, 5.53],
['Medium', 1.34, 3.44, 52.54, 13.76],
['Large', 1.83, 7.24, 269.19, 35.80],
['Monolith', 10.94, 162.93, 74.9, 79.90]];
chartPerformance('#incremental-changing-api', 'Build time in seconds (changing API)', data);
</script>
<p> </p>
<div id="incremental-keeping-api"></div>
<script>
var data = [
['Scenario', 'Gradle', 'Bazel', 'Maven (no clean)', 'Maven (local cache)'],
['Small', 0.93, 0.69, 7.97, 4.93],
['Medium', 1.04, 1.07, 52.86, 11.76],
['Large', 1.30, 3.01, 269.19, 35.80],
['Monolith', 10.87, 50.99, 74.9, 79.9]];
chartPerformance('#incremental-keeping-api', 'Build time in seconds (keeping API)', data);
</script>
<p>For the numbers in the first Maven column we did not run clean builds and it’s important to note that Maven is not reliable when not running clean. Even without clean, it also still executes most of the goals from scratch. Therefore you are taking the risk of a corrupt build for only a moderate performance gain. If you want a faster and yet reliable incremental build with Maven you can use Gradle Enterprise that provides a <a href="https://docs.gradle.com/enterprise/maven-build-cache/">local build cache for Maven</a>.</p>
<p>Both Gradle and Bazel build reliably without removing the output of a previous build (using <code class="language-plaintext highlighter-rouge">clean</code>). They both perform well in incremental scenarios with well modularized projects. Gradle has an edge over Bazel in this scenario.</p>
<p>An important result that is worth elaborating upon is that Gradle is 5-16 times faster than Bazel when executing an incremental change to the monolith. This is because Gradle has two levels to its process when dealing with incremental changes; one of which is very similar to the Bazel process. Both check if a certain action required by a module needs to be rebuilt at all depending on whether any of its input dependencies have changed. However, when Bazel needs to rerun a build action because of changed inputs, the build action for that module rebuilds everything. Gradle, in contrast, has a second layer to its process. With Gradle many build actions are deeply integrated with the toolchain and only what’s needed is rebuilt. For example, the Gradle compile build action only recompiles the source classes within a module that are affected by a change. Bazel recompiles all source classes. Gradle has many other tasks that rebuild incrementally and any Gradle plugin can make use of this, such as the Gradle Android plugin.</p>
<p>The Bazel answer to better incrementality is to break it into smaller modules and possibly rework the dependency graph. There is an appealing simplicity to this approach from the perspective of build system implementation but it comes with a price for end-users in terms of maintainability and applicability. It also reflects a lack of modelling from our point of view. This is discussed in the <a href="#build-authoring-and-maintenance">build authoring and maintenance</a> section.</p>
<h3 id="distributed-build--test-support">Distributed Build & Test Support</h3>
<p>Bazel supports a protocol for distributing a single build across multiple machines. Gradle Enterprise offers <a href="https://gradle.com/enterprise/releases/2020.2/#test-distribution-for-gradle-builds">distributed test execution</a> for Gradle, with Maven compatibility to come. Some CI products also offer functionality to distribute some parts of the build (e.g. <a href="https://www.jenkins.io/blog/2017/09/25/declarative-1/">parallel stages in Jenkins</a>). In contrast to the build cache, you often need to invest significantly into restructuring your code base to get substantial benefits from distributing your whole build. For most projects, test execution time is the dominant factor of overall build time. Gradle Enterprise test distribution accelerates existing test suites by distributing individual tests, without the need for significant restructuring.</p>
<p>Details of distributed testing and its performance impact will be covered in a separate article.</p>
<h2 id="build-authoring-and-maintenance">Build authoring and maintenance</h2>
<p>The performance section above demonstrated the differences in performance between Gradle and Bazel. However, raw performance is only a part of the story. Optimizing Bazel builds for optimal performance is a big topic on its own with significant consequences for build authoring and maintainability. In this section, we’ll dig deeper into these key considerations.</p>
<h3 id="difference-in-approaches">Difference in approaches</h3>
<p>Gradle and Bazel help users to achieve similar goals but they are actually quite different tools in terms of their design and what capabilities they provide.</p>
<p>Gradle aims to provide a complete solution for building software. It is highly optimized for JVM and Android projects, and deeply integrates with the toolchain to provide both performance and convenience. For example, in order to compile Java code, Gradle achieves incremental compilation by determining which classes to recompile based on sophisticated built-in dependency analysis.</p>
<p>Bazel provides an execution engine that delegates to external tools to do the actual work. For example, in the case of incremental compilation, Bazel simply delegates to <code class="language-plaintext highlighter-rouge">javac</code> and relies on fine-grained and explicit build files to compile the whole project incrementally.</p>
<p>This fundamental difference in approach between Gradle and Bazel goes well beyond the issue of incremental compilation. It has significant consequences to build authoring and maintenance, and affects a number of issues including build performance, system architecture, the depth of JVM support, and the breadth of support for non-JVM programming languages.</p>
<h3 id="the-granularity-of-build-files">The granularity of build files</h3>
<p>Gradle build files are typically defined on a project level, consisting of multiple packages of both production code and tests and related resources.</p>
<p>Bazel’s build files can be defined with different levels of granularity.</p>
<p>One can use the same approach as Gradle of a single build file per project. This is the pattern we observed in a number of open source projects using Bazel. This approach is most practical for most projects as we will elaborate on shortly. For these reasons, this is the granularity of Bazel build files that we selected for the performance comparisons.</p>
<p>The <a href="https://docs.bazel.build/versions/master/bazel-and-java.html#build-files">recommended approach</a> by Bazel for Java projects is to specify build files for each Java package as illustrated below.</p>
<p><img src="/images/gradle-vs-bazel/build-files.png" alt="Gradle vs Bazel build files" /></p>
<p>When using this approach in Bazel, the dependencies between packages are declared explicitly in build files. Cyclic dependencies between packages are not allowed and transitive dependencies have to be explicitly declared. This results in not only a large number of build files, but also build files that are verbose and require significant effort to maintain as the codebase evolves.</p>
<p>This recommended granularity of build files has a number of consequences that are important to understand before attempting such setup.</p>
<h4 id="performance-implications">Performance implications</h4>
<p>Package-level granularity is recommended for Bazel for optimal performance. It enables Bazel to perform a number of optimizations including compilation avoidance, parallel execution, and if the required infrastructure is available, also performing the work on remote machines.</p>
<p>Gradle integrates with the Java toolchain much more deeply, allowing optimizations such as incremental compilation and annotation processing to be effective without extra ceremony or structural constraints.</p>
<p>This explains why Gradle has a significant performance advantage in a number of scenarios that we tested. We would likely see improved results with Bazel if we used build file per Java package granularity. However, such an approach is not feasible for most existing projects, as we will explain shortly.</p>
<h4 id="architectural-implications">Architectural implications</h4>
<p>As a side note, we consider avoiding cyclic dependencies between packages to be a positive side effect of the Bazel model from the architectural point of view. We enthusiastically encourage well-modularized code bases. This is also important for the remote cache effectiveness. Gradle users can use tools such as ArchUnit or the Java Platform Module System (JPMS) to enforce this.</p>
<h4 id="migration-and-maintenance-cost-implications">Migration and maintenance cost implications</h4>
<p>The reality of almost every Java project is that cyclic dependencies exist between packages. The packages are also often not small and have many source classes that have dependencies on each other. Having small, fine-grained modules per package, is not just a question simply of how you configure your build logic; it often requires major refactoring, if not a complete rewrite.</p>
<p>Without seamless automated tooling, fine-grained build files in Bazel builds are very expensive to maintain. Internally at Google, such automation is provided by proprietary tools. For Bazel open source, users need to declare dependencies between packages manually. There have been various initiatives within the Bazel ecosystem to create such tooling for JVM builds. However, our investigation has found that all those projects are either archived or abandoned.</p>
<p>Gradle is fast without fine-grained build files. Fewer build files and less verbose build declarations means reduced maintenance and less ceremony when making changes to the code being built.</p>
<h3 id="jvm-vs-polyglot-support">JVM vs polyglot support</h3>
<p>The difference in approaches between the tools is also reflected in the depth of support in the JVM domain and the breadth of support of other languages.</p>
<p>Bazel provides an efficient execution engine that delegates to external commands and tools to do the actual work. It makes it easier for Bazel to provide support for more programming languages. As a result, Bazel includes official support for a number of non-JVM programming languages and platforms including C++, Python and Go.</p>
<p>On the other hand, Gradle is not just an efficient execution engine but a full-blown build tool. It offers the deepest level of support for JVM and Android. It makes it much easier to create and maintain high performing builds in these ecosystems. Still, Gradle provides official support for a number of non-JVM languages like C++ and Swift, and community plugins are available for several others including Python and Go.</p>
<h2 id="extensibility">Extensibility</h2>
<p>Complex builds rarely depend only on the built-in capabilities of the build tool. Typically users depend on extensions, with many being provided by the user community.</p>
<h3 id="custom-plugins--rules">Custom plugins / rules</h3>
<p>Gradle and Bazel provide mechanisms to share reusable pieces of custom build logic. The primary being <a href="https://docs.gradle.org/current/userguide/custom_plugins.html">custom plugins</a> and <a href="https://docs.bazel.build/versions/master/skylark/rules.html">rules</a> respectively. These are also the mechanisms used internally to provide built-in capabilities.</p>
<p>Bazel rules are implemented in Starlark, which is a language inspired by Python. In practice, many Bazel rules delegate to external tools and bash scripts.</p>
<p>Gradle plugins can be implemented in any JVM language (typically Java, Kotlin, or Groovy) and use rich Gradle APIs for common build needs. Using Gradle’s flexible DSL together with custom plugins allows Gradle users to create elegant and declarative build scripts. It can be leveraged for a very broad range of use cases and makes Gradle a powerful tool for even the most challenging integration scenarios.</p>
<h3 id="build-correctness">Build correctness</h3>
<p>One challenge with work avoidance optimizations such as incremental building and build caching is to ensure build correctness. The result of the build should be the same regardless of whether it was a full build or some work was avoided.</p>
<p>All of the steps in a build can be described as actions that produce outputs based on inputs. For example, Java compilation produces class files from source files (and has some additional inputs such as compiler options etc). In order for many optimizations to work correctly, the inputs and outputs must be correctly declared.</p>
<p>Both Gradle and Bazel provide built-in plugins/rules that are extensively tested for correctness. The main difference between the tools in this regard is in how correctness is ensured in custom build logic.</p>
<p>Bazel relies on <a href="https://github.com/bazelbuild/sandboxfs">sandboxing</a> to ensure that all inputs and outputs are correctly declared. If not, the custom rules will simply not work making correctness issues easier to discover. However, sandboxing incurs performance overhead and many Bazel users choose to disable it, <a href="https://github.com/bazelbuild/bazel/issues/8230">especially on MacOS</a>. Sandboxing is also <a href="https://github.com/bazelbuild/bazel/issues/5136">not supported on Windows</a>.</p>
<p>Gradle relies on users to correctly declare inputs and outputs in their custom build logic and plugins. Users can test their custom code with the <a href="https://docs.gradle.org/current/userguide/test_kit.html">TestKit</a> library. Many potential problems are caught automatically by the <a href="https://docs.gradle.org/current/userguide/java_gradle_plugin.html">Gradle Plugin Development Plugin</a>. Additionally, Gradle is becoming stricter in this regard. For example, Gradle 6.0 <a href="https://docs.gradle.org/6.0/release-notes.html#problems-with-tasks-called-out-during-build">deprecated</a> a number of potentially problematic behaviors that will be completely prevented in the next major version. Users can already opt in to fail their builds because of such issues with <a href="https://docs.gradle.org/current/userguide/command_line_interface.html#sec:command_line_warnings"><code class="language-plaintext highlighter-rouge">--warning-mode=fail</code></a> flag.</p>
<h2 id="dependency-management">Dependency management</h2>
<p>The vast majority of software projects require external dependencies. Managing them can be a cumbersome and error-prone process. Dependency management is a complex problem and a key feature of a build tool.</p>
<p>Modern JVM projects typically declare the dependencies they require and rely on the build tool to automate the process of obtaining the dependencies (and their dependencies) from an external repository, while managing version conflicts, caching, and many other considerations.</p>
<p>Both Gradle and Bazel provide support for dependency management workflows. Gradle has built-in support for both Maven and Ivy repositories. Bazel provides an <a href="https://github.com/bazelbuild/rules_jvm_external">official rule</a> with support only for Maven repositories, with some boilerplate code is required to use it. Both tools provide some form of conflict resolution and cache artifacts locally. Gradle provides more advanced transitive dependency management capabilities as explained below.</p>
<h3 id="dynamic-and-rich-version-declarations">Dynamic and rich version declarations</h3>
<p>Both Bazel and Gradle support declaring dependencies with static, deterministic, versions.</p>
<p>Gradle also supports <a href="https://docs.gradle.org/current/userguide/single_versions.html">version ranges</a> and <a href="https://docs.gradle.org/current/userguide/dynamic_versions.html#sub:declaring_dependency_with_dynamic_version">dynamic versions </a>to support various binary integration scenarios. In addition, <a href="https://docs.gradle.org/current/userguide/rich_versions.html">rich version declarations</a> allow defining dependency versions more precisely so that Gradle can make a better decision on which version to select and ultimately help users to avoid various dependency issues. Users can <a href="https://docs.gradle.org/current/userguide/dependency_locking.html">lock</a> version ranges and dynamic versions to ensure reproducible builds.</p>
<h3 id="transitive-dependency-management">Transitive dependency management</h3>
<p>Transitive dependency management is a complex topic and a simple version conflict resolution strategy is not enough. Small changes in declared dependencies often lead to unexpected compilation or even runtime errors as the transitive dependencies change.</p>
<p>Some of the common transitive dependency management issues that Gradle provides first-class solutions for are:</p>
<ul>
<li>A dependency version needs to be <a href="https://docs.gradle.org/current/userguide/dependency_downgrade_and_exclude.html">downgraded</a> because the project requires an older version of an external dependency than what is brought transitively and the default resolution strategy of selecting the newest version is not appropriate.</li>
<li>Related dependencies need to be <a href="https://docs.gradle.org/current/userguide/dependency_version_alignment.html">aligned</a> on the same version to work together correctly (for example different modules of the popular Jackson library).</li>
<li>Dependencies are mutually exclusive because they provide the same <a href="https://docs.gradle.org/current/userguide/dependency_capability_conflict.html">capabilities</a> (for example multiple logger implementations).</li>
<li>Dependencies provide different <a href="https://docs.gradle.org/current/userguide/variant_model.html">variants</a> that are optimized for a specific Java version (for example the Guava library with its Android and JRE flavors)</li>
<li>Dependencies provide <a href="https://docs.gradle.org/current/userguide/component_metadata_rules.html">wrong metadata</a>.</li>
<li>Dependencies require additional dependencies only for certain <a href="https://docs.gradle.org/current/userguide/component_capabilities.html">features</a>.</li>
</ul>
<p>In Bazel, none of the above use cases are supported. Users need to rely on explicit dependency declarations and excluding problematic versions to try to solve the issues above. For projects with non-trivial dependency graphs, this is expensive and hard to maintain. As the codebase evolves it will frequently lead to build and test failures that are hard to debug. If the built software is a library, it also pushes solving the same problems to consumers of the library.</p>
<h2 id="other-features">Other Features</h2>
<h3 id="build--test-facilities">Build & Test Facilities</h3>
<p>Gradle provides many conveniences for <a href="https://docs.gradle.org/current/userguide/building_java_projects.html">building and testing Java & JVM projects</a>. Bazel currently lacks built-in constructs for basic operations in JVM projects.</p>
<p>For example, Bazel users often rely on custom Python or shell scripts and handcrafted pom.xml files for publishing artifacts. In contrast, Gradle provides <a href="https://docs.gradle.org/current/userguide/publishing_maven.html">built-in support for publishing</a> with a high-level DSL and reasonable defaults.</p>
<p>Testing in Bazel is also more involved as there is no test detection. Instead, users have to explicitly declare test classes or test suites to compile and run. To run multiple tests, Bazel users often rely on a custom step in their build to generate JUnit test suites. In contrast, Gradle runs all tests that it detects by default. Additionally, JUnit 5 is not officially supported in Bazel at this time while in Gradle it can be enabled with a single line of declarative code.</p>
<h3 id="ide-support">IDE Support</h3>
<p>Gradle is supported out of the box in Intellij IDEA and an official plugin exists for Eclipse. Red Hat provides support for Java including Gradle support in Visual Studio Code. Bazel provides an official plugin only for IDEA. Currently, the <a href="https://ij.bazel.build/docs/bazel-support.html">Bazel support for IDEA</a> is on a “best-effort” basis for macOS and it is not supported on Windows. The official Eclipse support for Bazel is <a href="https://github.com/bazelbuild/eclipse">unmaintained</a>.</p>
<h3 id="spring-boot-support">Spring Boot Support</h3>
<p>Spring Boot Gradle Plugin is maintained by Spring Team. At this time only unofficial Spring Boot rules and examples exist for Bazel.</p>
<h3 id="kotlin-support">Kotlin Support</h3>
<p><a href="https://kotlinlang.org/docs/reference/using-gradle.html">Kotlin plugin for Gradle</a> is officially supported and maintained by JetBrains and the vast majority of Kotlin projects use Gradle. Bazel provides official <a href="https://github.com/bazelbuild/rules_kotlin">build rules for Kotlin</a>. Kotlin Multiplatform <a href="https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#metadata-publishing">makes use</a> of the built-in variant-aware dependency management that Gradle offers.</p>
<h2 id="popularity">Popularity</h2>
<p>The Gradle Build Tool is the most popular build tool for open-source JVM projects on GitHub. Most Java developers use it regularly according to the <a href="https://www.jetbrains.com/lp/devecosystem-2019/java/">State of Developer Ecosystem 2019</a> survey by Jetbrains. Gradle Build Tool downloads have doubled roughly every year since 2013 and it is now downloaded more than 15 million times per month. The Gradle Build Tool has a significantly larger plugin ecosystem (>4300 Gradle community plugins vs >120 Bazel rules maintained by either Google or the community).</p>
<h2 id="summary">Summary</h2>
<p>Both Gradle and Bazel are fast build systems with significant advantages compared to Maven, with Gradle Enterprise for Maven narrowing that gap. While we see a lot of merit in Bazel’s approach, particularly for extremely large monorepos like the one at Google, the analysis above suggests that Gradle is a better choice than Bazel for most JVM projects. Here is a summary of the key reasons:</p>
<ul>
<li>Gradle is faster than Bazel in almost all of the projects and scenarios measured including large monolith projects consisting of millions lines of code. This performance advantage arises from a number of factors including a deep integration with the Java toolchain. </li>
<li>Optimizing projects for Bazel comes at a significant cost for build authoring and maintenance due to fine-grained and verbose build file declarations, which is not the case with Gradle. Taking full advantage of Bazel’s performance optimizations is impractical for most existing projects, especially without additional tooling that does not exist for JVM builds.</li>
<li>Gradle provides more compelling features and conveniences for common use cases in JVM projects, including critical capabilities for dependency management and tool integration. Bazel currently lacks support for basic capabilities expected from a JVM build tool such as the ability to easily run all tests or publish artifacts.</li>
</ul>
<p>Whether you are using Gradle or Bazel to improve your developer productivity, choosing the faster build tool is not enough if not combined with a dedicated and continuous effort to keep your toolchain fast and reliable. This is why we recommend embracing the discipline of <a href="https://gradle.com/developer-productivity-engineering/">Developer Productivity Engineering</a> and a culture of developer productivity at all levels.</p>
Refining the Gradle issue backlog2020-06-09T00:00:00-04:00https://blog.gradle.org/stale-issue-backlogBenjamin Muskalla
<p>One of the biggest success factors of Gradle is its community. There is no day that goes by without users suggesting new features, finding and reproducing defects or opening a pull request with a contribution. We appreciate every single issue that users create. Thank you!</p>
<p>Working on a successful project like Gradle comes with challenges, be it the number of questions and contributions or deciding which aspect has the highest priority for the team. Even though we don’t like to accept it, we don’t have infinite capacity. We want our teams to be able to focus on producing the most value for our users. That means fixing the right bugs and implementing the best features. Given the diversity of our community, this is not an easy task to do. We want to find the right features/defects to do together with our users and not be paralyzed by the sheer volume of things we could potentially do.</p>
<p><img src="images/stale-issues/bt_issue_graph.png" alt="Open Issue trend over time for the Gradle Built Tool" /></p>
<h2 id="rationale">Rationale</h2>
<p>As part of our daily work, we triage new issues that you create, decide which ones should go into the next release, and check which issues are the most voted by the community. All while actually fixing existing issues and implementing new features. We’re quite happy and feel productive with our current workflow. One aspect we’re not very good at right now is managing the rest of our backlog. The sheer volume of ideas and contributions coming from the outside outweighs the number of tasks we can tackle within a reasonable timeframe. Given how many issues there are on our backlog, be it features, bugs or questions, it is really tough to find the time to go through all of them regularly. We need to determine which bugs are still reproducible in more recent versions of Gradle and which use cases have been implemented (maybe by just using one of the new features).</p>
<h2 id="approach">Approach</h2>
<p>As engineers, we sometimes struggle to close a defect even though it is not fixed. Several years ago, exactly that happened to myself. In a former life, my company had just hired our first product manager. She closed our whole backlog of 2000 issues…after her first week. Every. Single. Issue. I was shocked. I was raging. Looking back at it today, it was the best thing she could have done for that team back then to help focus on the important parts again. Over the following years, the team was able to focus again on the most important things and keep the backlog of tasks within the limits defined by the team. But don’t worry, our plan with the Gradle backlog will be less aggressive :)</p>
<p>Keeping this backlog in a state that we can derive our prioritization from is extremely important for the success of the Gradle project. You, as our community, play a vital part by helping us identify the important pieces and closing outdated issues. Not only that, we want to ensure that you know what the team is up to, what our priorities are and which features/defects simply fall out of scope given the bandwidth we have available. In order to help us with this, we’ve decided to introduce <a href="https://github.com/probot/stale">“Probot: Stale”</a>, a bot that looks at the activity (labels, comments, …) of issues to determine if they’re still active. If they’re inactive for a period of time (e.g. 2 years), it marks them with a <code class="language-plaintext highlighter-rouge">stale</code>, giving committers and members a chance to intervene. Once tagged with the <code class="language-plaintext highlighter-rouge">stale</code> label, the issues have a grace period of 21 days before they’re automatically closed.</p>
<p><img src="images/stale-issues/probot_stale.png" alt="Probot Stale" /></p>
<h2 id="what-does-this-mean-for-you">What does this mean for you?</h2>
<p>We understand that it can be hard to see older issues being closed automatically, especially if you opened the issue and care about it. But <a href="http://igorwiese.com/images/papers/Paper_BotSE_19.pdf">closing stale issues</a> does not mean we don’t care. In fact it is the opposite. This will help us to (re)discover the most important issues to prioritise them and you can help with this. So instead of just having your issues laying around unnoticed in the sheer mass of 2000+ unaddressed issues, you can now use the opportunity to get back to us about your concern.</p>
<p>It’s important to realize that these issues are marked and closed as stale. It’s not “this is invalid” or “we disagree that this is a cool feature”, it really is just “stale”. It means nobody in a long time had the bandwidth to look at it.</p>
<h2 id="going-forward">Going forward</h2>
<p>If an issue you care about is marked as stale or closed but it is still a concern for you: Please let us know and update the issue so we can reopen it for consideration. You will help us a lot and increase chances of progress, if you validate the issue against the latest Gradle release and if you make sure that you have a sample project attached that reproduces the bug or demonstrates the use case for the feature you miss.
We will be more open by saying “We don’t have the bandwidth”. Hopefully that will trigger a new conversation about the importance of the issue at hand - instead of letting it sit on the backlog, ignoring it for years to come.</p>
<p>Does this mean you should stop opening issues and PRs? No way! Please keep reporting issues you find, send your ideas/feature requests along and keep opening those PRs. We’re more than happy to review those and schedule them accordingly. This change enables us, the Gradle team, to regain focus on the aspects that provide the highest value for you, our community.</p>
Introducing file system watching2020-05-29T00:00:00-04:00https://blog.gradle.org/introducing-file-system-watchingStefan Wolf
<p><em>This is the first installment in a series of blog posts about incremental development–the part of the software development process where you make frequent small changes. We will be discussing upcoming Gradle build tool features that significantly improve feedback time around this use case.</em></p>
<p><em>This blog post has been updated after the feature has become production-ready in Gradle 6.7.</em></p>
<p>In Gradle 6.5, we have introduced an experimental feature called <em>file system watching</em> that significantly accelerates <a href="https://docs.gradle.org/current/userguide/more_about_tasks.html#sec:up_to_date_checks">incremental builds</a>.
When enabled, it allows Gradle to keep what it has learned about the file system in memory between builds instead of polling the file system on each build.
This significantly reduces the amount of disk I/O needed to determine what has changed since the previous build.</p>
<p>Since Gradle 6.7 this feature is ready for production use.
We are planning to enable the feature by default in a later release.</p>
<h2 id="how-does-it-work">How does it work?</h2>
<p>In order to determine whether Gradle needs to execute a task, it needs to check if any of its input and output files have changed since the last build.
The daemon stores this information about the file system till the end of the current build in memory in what we call the <em>virtual file system.</em></p>
<p>Without file system watching, the daemon does not know what happens to the file system in-between builds, and therefore it must discard all the collected information about the file system at the end of each build.
When file system watching is enabled, the daemon instructs the operating system to notify it about changes on disk.
Since the daemon knows what changed since the previous build, it can re-use the information in the virtual file system for all the unchanged locations, avoiding unnecessary disk I/O.</p>
<p>Gradle ships with the necessary integration for the recent versions of Linux, macOS, and Windows.
Other operating systems are not supported.</p>
<h2 id="file-system-watching-in-action">File system watching in action</h2>
<p>You can enable file system watching by passing the command line switch <code class="language-plaintext highlighter-rouge">--watch-fs</code> or setting a <a href="https://docs.gradle.org/current/userguide/build_environment.html#sec:gradle_configuration_properties">Gradle property</a>.
Once it’s enabled, the Gradle daemon will do the following:</p>
<ul>
<li>Start watching the file system for changes.</li>
<li>Keep the virtual file system during and between builds.</li>
</ul>
<p><strong>Note:</strong> we are enabling verbose logging to see the effect of file system watching.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./gradlew --watch-fs -Dorg.gradle.vfs.verbose=true :core:testClasses
Starting a Gradle Daemon
BUILD SUCCESSFUL in 24s
254 actionable tasks: 1 executed, 253 up-to-date
Received 11 file system events for current build while watching 1 hierarchies
Virtual file system retains information about 34797 files, 3845 directories and 128 missing files till next build
</code></pre></div></div>
<p>As you can see, the Gradle daemon learned quite a bit about the file system during this first build.
All that information is only available to the daemon used to run this particular build.</p>
<p>Now, let’s make a change to a source file.
Gradle will detect that change, and update the information about the source file in the virtual file system.
Then run another build:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./gradlew --watch-fs -Dorg.gradle.vfs.verbose=true :core:testClasses
Received 14 file system events since last build while watching 1 hierarchies
Virtual file system retained information about 34796 files, 3838 directories and 128 missing files since last build
BUILD SUCCESSFUL in 6s
254 actionable tasks: 3 executed, 251 up-to-date
Received 117 file system events for current build while watching 1 hierarchies
Virtual file system retains information about 34797 files, 3845 directories and 128 missing files till next build
</code></pre></div></div>
<p>Note that the daemon received some events since the last build and updated parts of the virtual file system.
Though most of the information remains intact.</p>
<h2 id="build-time-improvements">Build time improvements</h2>
<p>The practical impact of the feature depends on a number of factors, but in general it should result in a significant reduction of build time for <a href="https://docs.gradle.org/current/userguide/more_about_tasks.html#sec:up_to_date_checks">incremental builds</a>.
For example, in the <a href="https://github.com/gradle/santa-tracker-android/tree/agp-3.6.0-java">Santa Tracker Android project</a> we’ve seen the following improvement in the build time for a small change:</p>
<p><img src="images/introducing-file-system-watching/watch-fs-santa-tracker-linux.png" alt="Santa tracker linux FS watching improvements" /></p>
<p>In this instance we ran the same build many times, and on average we saw ~150ms (or 20%) faster execution times with file system watching enabled.</p>
<h2 id="try-out-file-system-watching">Try out file system watching</h2>
<p>If you like to see how your project benefits from file system watching, here is how you can try it out.</p>
<p>First, make sure you run Gradle 6.7 or later.
In order to <a href="https://docs.gradle.org/current/userguide/gradle_daemon.html#sec:daemon_watch_fs">enable file system watching</a>, you need to pass <code class="language-plaintext highlighter-rouge">--watch-fs</code> on the command-line.
Alternatively, add</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>org.gradle.vfs.watch=true
</code></pre></div></div>
<p>to the <code class="language-plaintext highlighter-rouge">gradle.properties</code> in the project directory or in the Gradle user home, so you don’t need to pass the command-line option on every build.
That’s it: the next build will run with file system watching enabled.</p>
<p><strong>Note:</strong> in Gradle 6.5 and 6.6 the <em>experimental</em> feature can be enabled via <code class="language-plaintext highlighter-rouge">org.gradle.unsafe.watch-fs=true</code>.</p>
<p>Keep in mind that you will only see performance improvements when consecutive builds on the same daemon have the feature enabled.</p>
<p>If you run into any problems, check out the <a href="https://docs.gradle.org/current/userguide/gradle_daemon.html#sec:daemon_watch_fs_troubleshooting">troubleshooting section</a> in the user manual.
If you still have problems contact us via the <a href="https://gradle-community.slack.com/app_redirect?channel=file-system-watching">Gradle community Slack</a>.</p>
<p>We also would love to hear about the improvements you saw in your build.
Please share your success stories in the <a href="https://gradle-community.slack.com/app_redirect?channel=file-system-watching">Gradle community Slack</a>.
If you want to benchmark your build, you can do it easily with <a href="https://github.com/gradle/gradle-profiler">Gradle profiler</a> by following the instructions in <a href="https://github.com/gradle/gradle-benchmark-base">this repository</a>.</p>
<p>This is not the end of the story regarding fast feedback for local incremental builds.
We have further improvements scheduled for the following Gradle releases!</p>
Plugin Portal Security CVE-2020-75992020-03-27T00:00:00-04:00https://blog.gradle.org/plugin-portal-updateSterling Greene
<h2 id="important-update-when-publishing-plugins-to-the-plugin-portal">Important update when publishing plugins to the Plugin Portal</h2>
<p>A security vulnerability was reported to us on March 4th, 2020. This problem could allow an authorized person to overwrite plugin artifacts on the Plugin Portal if they had access to the build logs that published the plugin. After a thorough investigation, we found no artifacts were overwritten for a malicious purpose.</p>
<p>In response, we’ve published a new version of the <code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code> plugin that contains an update to mitigate this security vulnerability.</p>
<p>Please upgrade <a href="https://plugins.gradle.org/plugin/com.gradle.plugin-publish"><code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code> plugin</a> to version <code class="language-plaintext highlighter-rouge">0.11.0</code>. Old versions of the <code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code> plugin will <strong>no longer work</strong>. If you do not publish plugins to the <a href="https://plugins.gradle.org/">Plugin Portal</a>, you do not need to do anything.</p>
<p>We also recommend that builds handling sensitive information (like publishing builds) do not run with elevated log levels (like <code class="language-plaintext highlighter-rouge">--debug</code> with Gradle) and are kept private to minimize the damage that can be done if sensitive information is exposed. You should also follow the best practices of your CI provider to avoid leaking sensitive information into build logs (<a href="https://docs.travis-ci.com/user/best-practices-security/#recommendations-on-how-to-avoid-leaking-secrets-to-build-logs">as an example, Travis CI</a>). Like other software, build maintainers and plugin authors need to keep in mind the types of information that may be logged.</p>
<p>This post is a summary of what we found and how we verified that artifacts served by the Plugin Portal were not changed. Continue reading if you’re interested in what we uncovered.</p>
<h3 id="discovery-of-the-vulnerability">Discovery of the Vulnerability</h3>
<p>On March 4th, 2020, we were notified about a security vulnerability with uploads to the Plugin Portal. The vulnerability could allow anyone with access to the log file from the build that published the plugins to overwrite the plugin’s artifacts when <code class="language-plaintext highlighter-rouge">info</code> level logging is enabled. This is an information disclosure vulnerability (<a href="https://cwe.mitre.org/data/definitions/532.html">CWE-532: Insertion of Sensitive Information into Log File</a>) for the Plugin Publish Plugin and is tracked by <a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-7599">CVE-2020-7599</a>.</p>
<p>Thanks to <a href="https://github.com/DanielThomas">Danny Thomas</a> from Netflix for reporting this issue to us.</p>
<p>When a plugin is published to the Plugin Portal, a <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html">pre-signed AWS S3 URL</a> is passed to the <code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code> plugin to upload artifacts. This URL was valid for 1 hour and could be re-used. By default, this URL was never shown to the user, but if the build ran with an elevated log level (<code class="language-plaintext highlighter-rouge">--info</code> or <code class="language-plaintext highlighter-rouge">--debug</code>), the pre-signed URL was captured in the build log file. With this URL, an attacker could then overwrite the plugin’s artifacts within that 1 hour window.</p>
<p>In general, it’s important that publicly facing builds be cautious with what is logged to their build output. Most CI systems attempt to filter out sensitive data from build logs, but in some cases, they may not hide everything; none of the CI providers filter these kinds of URLs as far as we know. Running your build with <code class="language-plaintext highlighter-rouge">debug</code> level logging can expose sensitive information about your infrastructure, passwords or internal web endpoints. This vulnerability was made possible with builds that ran with <code class="language-plaintext highlighter-rouge">info</code> level logging enabled.</p>
<h3 id="remediation-and-investigation">Remediation and Investigation</h3>
<p>After our investigation, we found no maliciously overwritten artifacts.</p>
<p>Once we became aware of the vulnerability, we deployed a change to limit the lifespan of the pre-signed URL. This greatly shortened the window of attack. Due to the way the <code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code> plugin works, the URL needs to remain valid for some amount of time to allow for all of the artifacts to be published.</p>
<p>We also investigated if any artifacts had been compromised. When publishing an artifact to the Plugin Portal, the client reports the SHA256 checksum of the artifact they intend to upload. We record that checksum, which allowed us to compare the original checksum against the checksum of each artifact in the S3 bucket. If the checksum of the artifact in the S3 bucket did not match the original checksum, this may indicate that the artifact was overwritten.</p>
<p>We audited all of the artifacts (over 190,000) available in the Plugin Portal for mismatched artifact hashes. We performed this comparison by downloading the contents of the S3 bucket and comparing the actual SHA256 checksums against our database. We initially identified over 9000 mismatches, but the vast majority of these artifacts were not accessible to users via the Plugin Portal. These artifacts were created for plugins that were deleted or that failed to publish all of their artifacts completely. Only 12 artifacts failed the checksum match and were served from the Plugin Portal. We investigated each of these to determine if they were compromised.</p>
<p>The artifacts consisted of</p>
<ul>
<li>non-executable artifacts, like source, javadoc and groovydoc that did not contain class files (4)</li>
<li>artifacts that were mismatched due to an earlier, unrelated, security investigation (2)</li>
<li>artifacts that were produced by a non-public build (2)</li>
<li>an artifact that was intentionally changed while investigating this vulnerability</li>
<li>a jar file that contained only timestamp differences when built locally</li>
<li>a pom file with no suspicious content</li>
<li>an invalid jar file</li>
</ul>
<p>None of the artifacts appear to be different in a meaningful way or could have been compromised for a malicious purpose.</p>
<p>We also reached out to several major cloud CI providers to help identify projects that may have been exposing the pre-signed URL in their build logs. We would like to thank the GitHub and CircleCI IR Teams for both being very proactive with their assistance.</p>
<h3 id="has-the-problem-been-patched-what-version-should-i-upgrade-to">Has the problem been patched? What version should I upgrade to?</h3>
<p>We have published a new version of the <code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code> plugin that reduces the log level for the pre-signed URL. Please upgrade <code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code> to version <code class="language-plaintext highlighter-rouge">0.11.0</code>. Previous versions of this plugin <strong>no longer work</strong> and will be rejected by the Plugin Portal.</p>
<p>These changes mitigate the exposure of the sensitive URL, but it’s still important that publicly facing builds be cautious with what is logged to their build output. Running Gradle with <code class="language-plaintext highlighter-rouge">--debug</code> will still expose the pre-signed URL. Internal JDK logging will log all HTTP requests–not just the artifact URLs.</p>
<p>Going forward, we will be making updates to the Plugin Portal to detect overwritten artifacts and to increase the security around the Gradle plugin ecosystem.</p>
<h3 id="i-cant-upgrade-is-there-anything-i-can-do">I can’t upgrade. Is there anything I can do?</h3>
<p>We’re requiring that everyone update to the latest version of the <code class="language-plaintext highlighter-rouge">com.gradle.plugin-publish</code> plugin. This version should work for anyone using Gradle 3.0 and above.</p>
<p>If you run into problems upgrading, please let us know with an <a href="https://github.com/gradle/gradle/issues/new/choose">issue</a>.</p>
<h3 id="for-more-information">For more information</h3>
<p>For security related issues, please email us at <a href="mailto:security@gradle.com">security@gradle.com</a>.</p>
<p>For non-security related issues, please open an issue on <a href="https://github.com/gradle/gradle/issues/new/choose">Github</a>.</p>
Smarter dependency downgrades2020-03-12T00:00:00-04:00https://blog.gradle.org/version-downgradeLouis Jacomet
<p>One of the biggest challenges when dealing with transitive dependencies is to keep their versions under control.
Popular libraries will show up as transitive dependencies in multiple places in your dependency graph.
And it is quite likely that the version information will be different on each path.</p>
<p>Through <a href="category/features">multiple blog posts</a>, you have learned that Gradle offers a rich feature set for expressing complex dependency requirements.
In this post, we will discuss why semantics matter when downgrading a dependency version.
And you will learn about the <em>strict version</em> feature of Gradle 6 that provides this semantic information and effectively gives you a <em>powerful and precise</em> tool for dealing with this complex issue.</p>
<p>In this post, we will use <a href="https://github.com/google/guava">Google’s Guava</a> <a href="guava">again</a> as an illustrating example because:</p>
<ul>
<li>It is a very popular library, <a href="https://mvnrepository.com/artifact/com.google.guava/guava">used by over 20,000 other libraries</a></li>
<li>It has a complex history in terms of API stability which results in sometimes difficult upgrades in large dependency graphs</li>
</ul>
<p>In the dependency view below, taken from <a href="https://scans.gradle.com/s/udugtvnlicvdg/dependencies?focusedDependency=WzAsMSwzOCxbMCwxLFswLDM4XV1d&focusedDependencyView=versions&toggled=W1swXSxbMCwxXSxbMCwxLFs0OV1dLFswLDEsWzBdXV0">this build scan</a>, we can see that many Guava versions are in play:</p>
<p><img src="/images/version-downgrade/guava-version-upgrade.png" alt="Extract of build scan showing conflicting Guava versions" /></p>
<p>The dependency declaration for this build is however quite simple:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dependencies {
implementation("org.optaplanner:optaplanner-core:7.24.0.Final")
implementation("com.spotify:folsom:1.5.0")
}
</code></pre></div></div>
<p>With only two direct dependencies, we already have a conflict between <em>four</em> Guava versions.
It will always resolve to <code class="language-plaintext highlighter-rouge">25.0-jre</code> in this example.
This is the result of Gradle accounting for all versions in a dependency graph and optimistically choosing the highest matching the constraints, <code class="language-plaintext highlighter-rouge">25.0-jre</code> in our example.</p>
<p>If we compare this Gradle project with a <a href="https://github.com/ljacomet/version-upgrade-blog/tree/master/maven-example">matching Maven project</a>, we get a somewhat different result.
With <a href="https://maven.apache.org/">Maven</a>, the <em>first</em> of the <em>shortest</em> path to a dependency will be used to determine the version.
This means that, in our example, the Guava version is effectively <em>order dependent</em> since both libraries have a direct dependency on Guava.
If <code class="language-plaintext highlighter-rouge">com.spotify:folsom:1.5.0</code> is declared first in a Maven POM file, Guava will be resolved at version <code class="language-plaintext highlighter-rouge">24.1-jre</code>.
However if <code class="language-plaintext highlighter-rouge">org.optaplanner:optaplanner-core:7.24.0.Final</code> comes first, Guava will be resolved at version <code class="language-plaintext highlighter-rouge">25.0-jre</code>.</p>
<h2 id="what-if-that-version-upgrade-is-a-problem">What if that version upgrade is a problem?</h2>
<p>Let’s pretend that bumping Guava to <code class="language-plaintext highlighter-rouge">25.0-jre</code> is an issue for Folsom because it relies on the <code class="language-plaintext highlighter-rouge">Files.fileTreeTraverser()</code> API from <code class="language-plaintext highlighter-rouge">24.1-jre</code>, <a href="https://github.com/google/guava/releases/tag/v25.0">removed in <code class="language-plaintext highlighter-rouge">25.0</code></a>.</p>
<p>Prior to Gradle 6, the most frequently used solutions for dealing with this problem were:</p>
<ul>
<li>Add <a href="https://docs.gradle.org/6.2.1/userguide/dependency_downgrade_and_exclude.html#sec:excluding-transitive-deps">an <code class="language-plaintext highlighter-rouge">exclude</code></a> on Guava on <code class="language-plaintext highlighter-rouge">org.optaplanner:optaplanner-core:7.24.0.Final</code></li>
<li>Add <a href="https://docs.gradle.org/6.2.1/userguide/dependency_downgrade_and_exclude.html#forced_dependencies_vs_strict_dependencies">a <code class="language-plaintext highlighter-rouge">resolutionStrategy.force</code></a> on Guava to version <code class="language-plaintext highlighter-rouge">24.1-jre</code></li>
</ul>
<p>Unfortunately, none of these solutions gave clear reasons for the version downgrade.
When excluding the dependency, it is not clear whether you meant that your usage of <code class="language-plaintext highlighter-rouge">org.optaplanner:optaplanner-core:7.24.0.Final</code> does not require Guava or if you only did it for its side effects on the resolved version.
If instead, you chose to force a particular version of the dependency, the information would not be available to the consumers of your library, which can be as simple as a different project in your multi project build, who would then be exposed to the same incompatibility you resolved.</p>
<p>The Maven situation is quite similar:</p>
<ul>
<li>An <code class="language-plaintext highlighter-rouge">exclude</code> has the same lack of semantics.</li>
<li>The use of <code class="language-plaintext highlighter-rouge">dependencyManagement</code> for transitive dependencies, like a Gradle <code class="language-plaintext highlighter-rouge">force</code>, is not applicable to consumers of your library.</li>
</ul>
<h2 id="a-meaningful-downgrade">A meaningful downgrade</h2>
<p>With Gradle 6, <a href="https://docs.gradle.org/6.2/userguide/rich_versions.html">rich version</a> provides an enhanced <em>strict version</em> declaration.
This version declaration has the following semantics:</p>
<ol>
<li>A <em>strict version</em> effectively takes precedence over all other dependency versions contributed by the subgraph of the project declaring the strict version.</li>
<li>A <em>strict version</em> effectively <em>rejects</em> all incompatible versions.</li>
</ol>
<p>So in our example, we would combine that version declaration with a <a href="https://docs.gradle.org/6.2/userguide/dependency_constraints.html#sec:direct-vs-transitive-deps">dependency constraint</a> to select Guava <code class="language-plaintext highlighter-rouge">24.1-jre</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dependencies {
constraints {
implementation("com.google.guava:guava") {
version {
strictly("24.1-jre")
}
because("Guava 25.0-jre removed APIs used by Folsom")
}
}
implementation("org.optaplanner:optaplanner-core:7.24.0.Final")
implementation("com.spotify:folsom:1.5.0")
}
</code></pre></div></div>
<h2 id="why-do-the-semantics-matter">Why do the semantics matter?</h2>
<p>In the example we took, simply combining two dependencies resulted in broken code.
We had to figure out which combination of dependency versions works <em>in the context of our own library</em>.
Then, we recorded our decision by adding a dependency constraint with a strict version.
If the library we are building is reused, it is important that this <em>decision is preserved</em> for future consumers.</p>
<p>With the pre Gradle 6 solutions, this information was lost.
Neither the <code class="language-plaintext highlighter-rouge">exclude</code> nor the <code class="language-plaintext highlighter-rouge">force</code> carried enough information to our consumers, which would most likely be bitten by the problem <em>again</em>.</p>
<p>The Maven solutions have the same downsides, with the same potential consequences.</p>
<p>With the <em>strict version</em> definition of Gradle 6, your consumers will know about the choice you made.
If any other of their dependencies causes a Guava update, the build will fail, indicating that your library has a <em>strict</em> requirement on Guava <code class="language-plaintext highlighter-rouge">24.1-jre</code>.
With the additional information provided by the <code class="language-plaintext highlighter-rouge">because</code> clause, these developers will know about the problem and already be on a path to find a solution of their own.
They can respect your choice or overrule it by defining their own <em>strict version</em>.</p>
<h2 id="best-practices-for-strict-versions">Best practices for strict versions</h2>
<p>Because of their semantics, strict versions should be added with the following best practices in mind:</p>
<ul>
<li>For reusable software libraries:
<ul>
<li>It is recommended to use a version range in the <code class="language-plaintext highlighter-rouge">strictly</code> part of the version when possible.
This gives more freedom to consumers of the library and enables them to find a solution without relying on another strict version definition.</li>
<li>Provide a <code class="language-plaintext highlighter-rouge">prefer</code> version which can be used when consumers do not care.</li>
</ul>
</li>
<li>For applications, a fixed version in the <code class="language-plaintext highlighter-rouge">strictly</code> part is the simplest and most direct option.</li>
<li>In all cases, make sure to <a href="https://docs.gradle.org/6.2/userguide/declaring_dependencies.html#sec:documenting-dependencies">document your decision with a <code class="language-plaintext highlighter-rouge">because</code></a>.</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>A consequence of libraries reuse is that version conflicts are unavoidable, especially for popular ones.
Sometimes it is necessary to make an explicit version choice for a specific combination of libraries.
The <em>strict version</em> concept of Gradle 6 allows you to make this choice and to preserve it for your consumers.</p>
<p>While the Maven resolution mechanism sounds simpler at first, we showed that <a href="https://gradle.org/maven-vs-gradle/#dependency-management">it has semantic issues</a> as the dependency graph grows.
A solution that makes sense for your library loses all meaning when consumed.</p>
<p>On the other hand, Gradle, by associating semantics with version declaration, has a consistent resolution model, which allows developers to clearly express the choices they made and document their reasoning.</p>
<p>These choices are then available when a library is consumed by others, giving the opportunity to consumers to better respect, or overrule and document, the choices made by the library developers.</p>
Verifying Gradle Wrappers with GitHub Actions2020-02-06T00:00:00-05:00https://blog.gradle.org/gradle-wrapper-checksum-verification-github-actionJonathan Leitschuh
<p><img src="images/verify-wrapper-action/wrapper-validation-status-check.png" alt="Pull Request Status Check with new 'Validate Gradle Wrapper / Validation' successful status" /></p>
<p>We are proud to announce the release of the new
<a href="https://github.com/marketplace/actions/gradle-wrapper-validation">Gradle Wrapper Validation GitHub Action</a>.</p>
<h2 id="gradle-wrapper-in-open-source">Gradle Wrapper in Open Source</h2>
<p>The <a href="https://docs.gradle.org/current/userguide/gradle_wrapper.html"><code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code></a>
is a binary blob of executable code that is checked into nearly
<a href="https://github.com/search?l=&q=filename%3Agradle-wrapper.jar&type=Code">2.8 Million GitHub Repositories</a>.</p>
<p>Searching across GitHub you can find many pull requests (PRs) with helpful titles like ‘Update to Gradle xxx’.
Many of these PRs are contributed by individuals outside of the organization maintaining the project.</p>
<p>Maintainers are grateful for these kinds of contributions as it takes an item off of their backlog.
But there are security implications of accepting changes to the Gradle Wrapper binary from external contributors that may not be apparent.
An attacker could take advantage of the trust the open source community has by
hiding malicious code inside the Gradle Wrapper.
A malicious <code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code> could download, install and execute arbitrary code while otherwise behaving like
a completely normal <code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code>.</p>
<p>Such an attack could be easily missed as the diff to the <code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code> looks like this.</p>
<p><img src="images/verify-wrapper-action/binary-wrapper-file-diff.png" alt="Image of a GitHub Diff of Gradle Wrapper displaying text 'Binary file not shown.'" /></p>
<h2 id="verifying-the-gradle-wrapper">Verifying the Gradle Wrapper</h2>
<p>We have created a
<a href="https://github.com/marketplace/actions/gradle-wrapper-validation">simple GitHub Action</a>
that can be applied to any GitHub repository.
This action will verify that any and all <code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code> files in the repository match an
<a href="https://gradle.org/release-checksums/">official SHA-256 checksum</a>.
If any file does not match, the action will fail.</p>
<p>Additionally, the action will detect any <a href="https://en.wikipedia.org/wiki/Homoglyph">homoglyph</a> variants of a file named <code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code>.
The goal is to prevent difficult to spot homoglyph attacks,
like renaming the <code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code> file to <code class="language-plaintext highlighter-rouge">gradlе-wrapper.jar</code> (which uses a Cyrillic <code class="language-plaintext highlighter-rouge">е</code> instead of <code class="language-plaintext highlighter-rouge">e</code>).</p>
<h2 id="securing-your-project">Securing Your Project</h2>
<p>GitHub actions are free to use for open-source and are automatically enabled by default on almost all repositories.
You can find out more about how to add this action to your GitHub repository
<a href="https://github.com/marketplace/actions/gradle-wrapper-validation#usage">here</a>.</p>
<h3 id="existing-users">Existing Users</h3>
<p>This action has already been contributed to some of the most popular Gradle based projects on GitHub including but not limited to:</p>
<ul>
<li><a href="https://github.com/ReactiveX/RxJava">ReactiveX/RxJava</a></li>
<li><a href="https://github.com/ReactiveX/RxAndroid">ReactiveX/RxAndroid</a></li>
<li><a href="https://github.com/junit-team/junit5-samples">junit-team/junit5</a></li>
<li><a href="https://github.com/JakeWharton/butterknife">JakeWharton/butterknife</a></li>
<li><a href="https://github.com/google/flexbox-layout">google/flexbox-layout</a></li>
<li><a href="https://github.com/ktorio/ktor">ktorio/ktor</a></li>
<li><a href="https://github.com/facebook/fresco">facebook/fresco</a></li>
<li><a href="https://github.com/facebook/flipper">facebook/flipper</a></li>
<li><a href="https://github.com/mockito/mockito">mockito/mockito</a></li>
<li><a href="https://github.com/jhipster/generator-jhipster">jhipster/generator-jhipster</a></li>
<li><a href="https://github.com/apache/groovy">apache/groovy</a></li>
<li><a href="https://github.com/apache/lucene-solr">apache/lucene-solr</a></li>
</ul>
<p>And <a href="https://github.com/search?l=YAML&q=gradle%2Fwrapper-validation-action&s=indexed&type=Code">many more</a>!</p>
<h3 id="reporting-failures">Reporting Failures</h3>
<p>If this GitHub action fails because a <code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code> doesn’t match one of our published SHA-256 checksums,
we highly recommend that you reach out to us at <a href="mailto:security@gradle.com">security@gradle.com</a>.</p>
<p>If you’re curious and want to explore what the differences are between the <code class="language-plaintext highlighter-rouge">gradle-wrapper.jar</code> in your possession
and one of our valid releases, you can compare them using this online utility: <a href="https://try.diffoscope.org/">DiffScope</a>.</p>
<h2 id="resources">Resources</h2>
<p>To learn more about verifying the Gradle Wrapper JAR locally (ie. without using GitHub Actions), see our
<a href="https://docs.gradle.org/current/userguide/gradle_wrapper.html#wrapper_checksum_verification">documentation on the topic</a>.</p>
Why libraries like Guava need more than POMs2020-01-22T00:00:00-05:00https://blog.gradle.org/guavaJendrik Johannes
<p>More than 10 years ago, a new Java collections library was released by Google.
This library, now known as <a href="https://github.com/google/guava">Google Guava</a>, would gain a lot of traction over the months and years and is possibly the most used Java library in production code today.</p>
<p>Due to the widespread adoption of Guava, many other libraries depend on it today.
Chances are high that you will find it on the classpath of any reasonably large Java project through transitive dependencies, even if it is not used directly.
With more and more code depending on such a widely used library, the potential for conflicts increases, adding to a project’s <a href="https://blog.gradle.org/avoiding-dependency-hell-gradle-6">dependency hell</a>.
<!--break--></p>
<p>Consider the following harmless-looking dependency declaration block:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dependencies {
implementation("com.google.guava:guava:28.0-jre")
implementation("org.codehaus.plexus:plexus-container-default:2.1.0")
implementation("com.google.api-client:google-api-client:1.30.7")
}
</code></pre></div></div>
<p>We would expect to end up with a JRE (and not an Android) variant of Guava and we would expect the build tool to inform us if there are other suspicious conflicts on our classpath.
Let’s take a look at a <a href="https://scans.gradle.com/s/jpi72mv2zoow2/dependencies?toggled=W1swXSxbMCwxXSxbMCwxLFs0OF1dLFswLDEsWzM1XV0sWzAsMSxbNDNdXV0">build scan</a> showing the dependencies graph:</p>
<p><img src="/images/gradle-6-dm/guava-scan.png" alt="Build scan showing a dependency graph with conflicting Guava versions" /></p>
<p>If we look closely, we can observe some unexpected things:
Why was <code class="language-plaintext highlighter-rouge">guava:28.0-jre</code>, replaced with <code class="language-plaintext highlighter-rouge">guava:28.1-android</code> without warning?
Why is there a <code class="language-plaintext highlighter-rouge">google-collections</code> dependency – wasn’t that the same as Guava?
Why do I need <code class="language-plaintext highlighter-rouge">j2objc-annotations</code> on my runtime classpath?
And what is this weird <code class="language-plaintext highlighter-rouge">9999.0-empty-to-avoid-conflict-with-guava</code> dependency?</p>
<p>To understand this, we will discuss the dependency management challenges that appeared during the evolution of Guava and how they were handled.
In the end, we will show how troubles can be avoided using Gradle Module Metadata.</p>
<h2 id="naming-things-is-hard">Naming things is hard</h2>
<p>The dependency management troubles started early for what was previously known as the Google Collections Library.
<code class="language-plaintext highlighter-rouge">com.google.collections:google-collections:1.0</code> are the coordinates under which the final release of the <a href="https://code.google.com/archive/p/google-collections/">Google Collections Library</a> was published to the <a href="https://mvnrepository.com/artifact/com.google.collections/google-collections/1.0">Maven central repository back in 2009</a>.
In 2010, the first stable version of <a href="https://mvnrepository.com/artifact/com.google.guava/guava">Guava</a>, <code class="language-plaintext highlighter-rouge">com.google.guava:guava:10.0</code>, was released including all of Google Collections along with other utilities – replacing the Google Collections Library.</p>
<p>As a result of this “rename” from <code class="language-plaintext highlighter-rouge">google-collections</code> to <code class="language-plaintext highlighter-rouge">guava</code>, the dependency management engines of Gradle and Maven were no longer able to detect conflicts between versions of Google Collections and Guava.
Such an unhandled conflict led to two jars on the classpath containing different versions of the Google Collections classes.
In the case where build authors discovered the conflict by chance, they had to <a href="https://stackoverflow.com/a/13191483/4274701">manually exclude <code class="language-plaintext highlighter-rouge">google-collections</code> as transitive dependency</a> or, in Gradle, register a <a href="https://docs.gradle.org/6.0/userguide/resolution_rules.html#sec:module_replacement">replacement rule</a>.</p>
<h2 id="alphabetic-versioning-trouble">Alphabetic versioning trouble</h2>
<p>When version <code class="language-plaintext highlighter-rouge">22.0</code> of Guava was released in May 2017, Guava had moved from Java 6 to Java 8. Android, however, was still stuck on Java 6.
Without a change, Android users would have been stuck on old versions forever.
Therefore, Guava <a href="https://github.com/google/guava/wiki/Release22#new-android-version">started shipping a separate Android variant</a> stripped of all Java 8 specific functionality.</p>
<p>The two variants were released using the same <code class="language-plaintext highlighter-rouge">com.google.guava:guava</code> coordinates with two different version strings: <code class="language-plaintext highlighter-rouge">22.0</code> and <code class="language-plaintext highlighter-rouge">22.0-android</code>.
After a longer discussion on <a href="https://github.com/google/guava/issues/2914">GitHub</a> and a public <a href="https://docs.google.com/document/d/1NYGbfz56C0Oh4IGymXjeQUVK4FcRiqDbpc4vGLnDMrY">GoogleDoc</a>, the versioning pattern changed with 23.1 to <code class="language-plaintext highlighter-rouge">23.1-jre</code> and <code class="language-plaintext highlighter-rouge">23.1-android</code>.
Instead of using a different classifier or coordinates for the different variants, using different versions allowed the dependency conflict resolution in Gradle and Maven to detect a conflict and select only one of the two variants.
(The <code class="language-plaintext highlighter-rouge">-jre</code> suffix was introduced to make sure that the <code class="language-plaintext highlighter-rouge">-jre</code> version is always considered higher than the <code class="language-plaintext highlighter-rouge">-android</code> version by Maven because <code class="language-plaintext highlighter-rouge">j</code> beats <code class="language-plaintext highlighter-rouge">a</code> in alphabetical order.)</p>
<h2 id="j-not-always-beats-a">J (not always) beats A</h2>
<p>While the introduction of the <code class="language-plaintext highlighter-rouge">-jre</code> suffix solved some issues for Maven users, problems still remained for Gradle and Maven users if multiple versions of Guava were involved.</p>
<p>Looking again at our <a href="https://scans.gradle.com/s/jpi72mv2zoow2/dependencies?toggled=W1swXSxbMCwxXSxbMCwxLFs0OF1dLFswLDEsWzM1XV0sWzAsMSxbNDNdXV0">initial example</a>,
the actual version <em>and</em> the variant are both encoded in the version strings: <code class="language-plaintext highlighter-rouge">28.0-jre</code> and <code class="language-plaintext highlighter-rouge">28.1-android</code>.
The build tools cannot know how to use this information.
Gradle, looking at the full version string, picks the higher version: <code class="language-plaintext highlighter-rouge">28.1-android</code>.
This is a version without the Java 8 specific classes, which might very well break code that relies on those classes.
The best solution would probably be to pick <code class="language-plaintext highlighter-rouge">28.1-jre</code> as it satisfies both requests: <code class="language-plaintext highlighter-rouge">28.1</code> (which is assumed to be compatible with <code class="language-plaintext highlighter-rouge">28.0</code>) and <code class="language-plaintext highlighter-rouge">jre</code> (which is compatible with <code class="language-plaintext highlighter-rouge">android</code>).
However, requesting a version and a variant independently cannot be modelled with POM metadata.</p>
<h2 id="annoying-annotation-libraries">Annoying annotation libraries</h2>
<p>Being aware of its wide-spread use, Guava’s code is mostly self contained, avoiding additional dependencies.
Still, over time, Guava added some dependencies to annotation libraries, such as <code class="language-plaintext highlighter-rouge">com.google.code.findbugs:jsr305</code> or <code class="language-plaintext highlighter-rouge">com.google.errorprone:error_prone_annotations</code>, required at compile time.</p>
<p>Many <a href="https://github.com/google/guava/issues/2824">users are annoyed</a> by these dependencies as they are also present on the runtime classpath.
The annotation library dependencies are only there to avoid compile time warnings if the annotations on Guava’s classes are inspected by the Java compiler.
At runtime, the annotations are not touched and thus the annotation library jars are not needed.
However, each compile scope dependency defined in a POM is automatically present on the runtime classpath and there is no concept to declare a dependency <em>only</em> for compile time.</p>
<h2 id="duplication-troubles">Duplication troubles</h2>
<p>In September 2018, one interface – <code class="language-plaintext highlighter-rouge">ListenableFuture</code> – was copied from Guava into a separate module – <code class="language-plaintext highlighter-rouge">com.google.guava:listenablefuture:1.0</code> – to <a href="https://groups.google.com/forum/#!topic/guava-announce/Km82fZG68Sw">allow Android developers to use it in their APIs</a>, without depending on all of Guava.
To keep Guava self contained, the development team decided to copy that interface instead of completely moving it out of Guava.
Instead, they published an empty version of the ListenableFuture module – <code class="language-plaintext highlighter-rouge">com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava</code> – on which Guava depends now.
This is tricking Gradle’s dependency management engine to always use the empty <code class="language-plaintext highlighter-rouge">9999.0-empty-to-avoid-conflict-with-guava</code> version of the ListenableFuture module when Guava is used.
The real version <code class="language-plaintext highlighter-rouge">1.0</code>, which contains the duplication of the <code class="language-plaintext highlighter-rouge">ListenableFuture</code> interface, is not picked then.</p>
<p>While this seems to be an ingenious approach, and it is for many cases of Android development where Gradle is used, there are also <a href="https://github.com/google/guava/issues/3320">problems with it</a>.
JVM build authors, who only use Guava, complained that they always have an additional empty jar on their classpath even if there is no conflict.
In Maven, the approach only works in certain setups, since Maven does not necessarily pick the highest version, but the closest – i.e. if <code class="language-plaintext highlighter-rouge">com.google.guava:listenablefuture:1.0</code> is discovered first in the dependency graph, it will be picked over the empty version.</p>
<h2 id="when-you-know-what-you-want-but-you-cannot-express-it">When you know what you want but you cannot express it</h2>
<p>If you now think that the Guava team should have done a better job with all those troubles, you are mistaken.
In fact, as you can see from the linked discussions, the team was very concerned with each decision taken.</p>
<p>The root cause of the troubles is that the POM metadata model is not expressive enough to convey the required information.
As Guava and other libraries have shown over the past decade, there is a need to express more information in metadata to solve many common use cases.
As an answer to this need, we developed the <a href="gradle-metadata-1.0">Gradle Module Metadata</a> format.</p>
<p>With Gradle Module Metadata, the problems described in this post can be addressed:</p>
<ul>
<li><strong>Renaming to Guava</strong>
Gradle Module Metadata offers the modelling concept of <a href="addressing-logging-complexity-capabilities">capabilities</a>.
With it, a module can express that it provides an alternative implementation of something implemented by another module.
<br />
Each version of Guava could declare that it provides the <code class="language-plaintext highlighter-rouge">com.google.collections:google-collections</code> capability and Gradle then would detect the conflict if both Google Collections and Guava are part of the dependency graph.</li>
<li><strong>Publishing more variants</strong>
With Gradle Module Metadata, each module has arbitrarily many <a href="https://docs.gradle.org/current/userguide/variant_model.html">variants</a>.
Each variant can point to different artifacts (jars) and can have different dependencies.
A variant is identified by a number of attributes including, in the case of Java libraries, the <code class="language-plaintext highlighter-rouge">org.gradle.jvm.version</code> attribute.
<br />
Guava could publish variants for different Java versions, JRE (Java 8) and Android (Java 6/7), in one module.
Gradle will then select the right variant based on the Java version used.</li>
<li><strong>Compile-time only dependencies</strong>
A module published with Gradle Module Metadata explicitly defines <em>runtime</em> and <em>compile time (api)</em> variants where each of them defines dependencies independently.
<br />
Using this flexible <a href="https://docs.gradle.org/current/userguide/java_library_plugin.html#sec:java_library_separation">API and implementation separation</a>, Guava could add annotation library dependencies to the compile time variant only preventing them from leaking onto the runtime classpath.</li>
<li><strong>Copying ListenableFuture to a second module</strong>
The <a href="addressing-logging-complexity-capabilities">capabilities</a> concept offered by Gradle Module Metadata, that can be used to solve the renaming issue, can also be used here.
<br />
Guava could declare that it provides the <code class="language-plaintext highlighter-rouge">com.google.guava:listenablefuture</code> capability which is enough for Gradle to detect a conflict with the real <code class="language-plaintext highlighter-rouge">listenablefuture</code> module if it appears in the dependency graph.
In the most common case, where there is no conflict, the capability will have no effect. The dependency on the empty <code class="language-plaintext highlighter-rouge">listenablefuture</code> module can be removed.</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>In this blog post we’ve taken you on a journey through the history of Guava, as an example of an evolving library that is widely used.
As you might have experienced yourself, for such libraries, the chance for version and variant conflicts are high.
But with the right amount of metadata published, these conflicts can be detected, and resolved, by build tools.</p>
<p>As we have shown, Guava’s developers are very aware of this and did not take decisions lightly.
However, they were limited by the expressiveness of the POM format multiple times.
Despite their best efforts, and the application of multiple tricks, many build authors still face issues with undetected and unresolved conflicts involving Guava.</p>
<p>To improve the situation in the future, we created a <a href="https://github.com/google/guava/pull/3683">pull request</a> that proposes publishing of Gradle Module Metadata for Guava.
For already published versions of Guava, or other libraries, Gradle allows you to write a <a href="https://docs.gradle.org/current/userguide/component_metadata_rules.html">component metadata rule</a> to add missing metadata.
We have written <a href="https://github.com/jjohannes/missing-metadata-guava/blob/master/src/main/java/de/jjohannes/gradle/missingmetadata/guava/GuavaMetadataRule.java">such a rule</a> for published versions for Guava and provide it as a Gradle plugin.
If you apply <a href="https://plugins.gradle.org/plugin/de.jjohannes.missing-metadata-guava">this plugin</a> to your build, you can explore what we have described in the blog post yourself:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>plugins {
id("de.jjohannes.missing-metadata-guava") version "0.1"
}
</code></pre></div></div>
<p>If we add the plugin to the example from the beginning, you can observe, for instance, the <a href="https://scans.gradle.com/s/tngwlr7jgzl72/dependencies?toggled=W1swXSxbMCwxXSxbMCwxLFszM11dLFswLDEsWzQwXV0sWzAsMSxbMzVdXV0">reduced runtime classpath</a>
and that the <a href="https://scans.gradle.com/s/tngwlr7jgzl72/dependencies?focusedDependency=WzAsMSwzMyxbMCwxLFszM11dXQ&toggled=W1swXSxbMCwxXSxbMCwxLFszM11dLFswLDEsWzQwXV0sWzAsMSxbMzVdXV0">Java 8 variant of Guava was chosen</a>,
which provides the <a href="https://scans.gradle.com/s/tngwlr7jgzl72/console-log#L32">guava-28.1-jre.jar</a> despite the selection of the <code class="language-plaintext highlighter-rouge">28.1-android</code> version.</p>
<p>If you are developing libraries yourself, or know about libraries facing similar issues, feel free to <a href="https://github.com/gradle/gradle/issues/11528">reach out to us</a>.
We are happy to explore improvements by publishing Gradle Module Metadata!</p>
Introducing flaky test mitigation tools2020-01-13T00:00:00-05:00https://blog.gradle.org/gradle-flaky-test-retry-pluginEric Wendelin
<p>This post introduces a <a href="https://github.com/gradle/test-retry-gradle-plugin">new Gradle plugin</a> and build scans improvements aimed at mitigating your flaky tests.</p>
<p>Flaky tests disrupt software development cycles by blocking CI pipelines and causing unnecessary failure investigations.
Unhealthy teams live by re-running builds, sometimes several times, to get changes through.
Martin Fowler has <a href="https://martinfowler.com/articles/nonDeterminism.html">pointed words about non-deterministic tests</a> that are worth a read.</p>
<p>To eliminate this parasite from your organization you have to identify, prioritize, and fix your flaky tests.</p>
<h2 id="mitigating-flaky-tests">Mitigating flaky tests</h2>
<p>There are a number of clever heuristics that help identify flaky tests.
You could run some static analysis to prove that a test failure could theoretically not cause a given test failure.
You could count the number of “flips” from failed to passed and some threshold that over which a test is considered flaky.</p>
<p>These heuristics work, usually, but they are really, really hard to get right.
A big problem arises when your flaky test detection methodology is itself flaky — people do not trust the system and they resolve to rerun-and-suffer mode until they get the result they want.</p>
<p>Re-running tests in the same execution environment is a way of identifying a flaky test beyond reasonable doubt.
It’s no wonder so many teams and libraries incorporate this simple strategy.
It’s even been built directly into Maven Surefire and Failsafe.</p>
<p>This is why we’ve developed the <a href="https://github.com/gradle/test-retry-gradle-plugin">Test Retry Gradle Plugin</a> that retries failed tests for the purposes of mitigating test flakiness.</p>
<h2 id="new-test-retry-gradle-plugin">New Test Retry Gradle Plugin</h2>
<p>You can use this Gradle configuration to retry tests and optionally fail the build on flakiness:</p>
<div class="language-groovy exampleblock multi-language-sample highlighter-rouge" data-lang="groovy"><div class="highlight"><pre class="highlight"><code><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s1">'org.gradle.test-retry'</span> <span class="n">version</span> <span class="s1">'1.0.0'</span>
<span class="o">}</span>
<span class="n">test</span> <span class="o">{</span>
<span class="n">retry</span> <span class="o">{</span>
<span class="n">failOnPassedAfterRetry</span> <span class="o">=</span> <span class="kc">true</span>
<span class="n">maxFailures</span> <span class="o">=</span> <span class="mi">42</span>
<span class="n">maxRetries</span> <span class="o">=</span> <span class="mi">1</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<div class="language-kotlin exampleblock multi-language-sample highlighter-rouge" data-lang="kotlin"><div class="highlight"><pre class="highlight"><code><span class="nf">plugins</span> <span class="p">{</span>
<span class="nf">id</span><span class="p">(</span><span class="s">"org.gradle.test-retry"</span><span class="p">)</span> <span class="n">version</span> <span class="s">"1.0.0"</span>
<span class="p">}</span>
<span class="n">tasks</span><span class="p">.</span><span class="nf">test</span> <span class="p">{</span>
<span class="nf">retry</span> <span class="p">{</span>
<span class="n">failOnPassedAfterRetry</span><span class="p">.</span><span class="k">set</span><span class="p">(</span><span class="k">true</span><span class="p">)</span>
<span class="n">maxFailures</span><span class="p">.</span><span class="k">set</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span>
<span class="n">maxRetries</span><span class="p">.</span><span class="k">set</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>There are 4 especially neat aspects to this plugin:</p>
<ol>
<li>No test source changes are required. This allows <em>proactive detection</em> of new flaky tests!</li>
<li>You can control whether your build fails or passes when flakiness is encountered using <code class="language-plaintext highlighter-rouge">failOnPassedAfterRetry</code>. This means that you can adopt this plugin to detect flaky tests without silencing them.</li>
<li>You can prevent retrying in a test run after a discrete number of tests fail using <code class="language-plaintext highlighter-rouge">maxFailures</code>. If your build encounters many failures, it’s likely there is a major problem causing many tests to fail and retrying is a waste of resources.</li>
<li>Tests are retried at method-level or finer where possible — no rerunning whole classes of tests!</li>
</ol>
<h3 id="supported-environments-of-gradle-test-retry-plugin">Supported environments of Gradle Test Retry Plugin</h3>
<p>The Test Retry Plugin supports Gradle 5.0 and later out-of-the-box with the following test frameworks:</p>
<ul>
<li>JUnit4</li>
<li>JUnit Platform (JUnit 5)</li>
<li>Spock</li>
<li>TestNG</li>
</ul>
<p>In some cases, passing upstream tests or downstream tests (when using <code class="language-plaintext highlighter-rouge">@Test(dependsOn = {})</code>, for example) must be re-executed by the plugin to ensure correctness where a flaky test depends on state from tests it depends on.</p>
<p>For more information about supported frameworks and retry mechanics, please see the <a href="https://github.com/gradle/test-retry-gradle-plugin/blob/master/README.adoc">Test Retry Gradle Plugin docs</a>.</p>
<h2 id="how-flaky-tests-are-reported">How flaky tests are reported</h2>
<p>We chose to report all discrete executions of flaky tests to maximize compatibility with existing test reports and IDEs. Therefore a flaky test report might look like this:</p>
<p><img src="/images/gradle-test-retry-plugin/gradle-reports-test-retry-reporting.png" alt="JUnit report with multiple test executions" /></p>
<p>We are working to make flaky a first-class test outcome in more tools you use daily to clarify reporting.
<a href="https://github.com/gradle/test-retry-gradle-plugin/blob/master/README.adoc">The docs</a> have more details about how flaky tests are reported in logs, test reports, and popular IDEs.</p>
<p>In good news, <a href="https://scans.gradle.com/">build scans</a> already report flaky tests in a clear and accurate way!</p>
<h2 id="flaky-tests-support-for-build-scans">Flaky tests support for build scans</h2>
<p>Gradle and Maven build scans now report a test as flaky when it is executed multiple times in the same build with both a PASSED and FAILED result, in any order.
This allows build scans to report a flaky test using in-build retry mechanisms such as Maven’s <a href="https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html">rerun failing tests</a> options such as <code class="language-plaintext highlighter-rouge">-Dsurefire.rerunFailingTestsCount=2</code> or your custom retry mechanism.</p>
<p><img src="/images/gradle-test-retry-plugin/build-scan-with-flaky-tests.png" alt="Build scan with flaky tests" /></p>
<p>Identifying and silencing flaky tests only treats symptoms. There are likely some real concurrency or performance issues that cause suffering for your customers — you must analyze flaky test executions to fix them.
You can read more in my blog post about <a href="https://gradle.com/blog/flaky-tests/">analyzing flaky tests in Maven and Gradle builds</a>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>We hope the new <a href="https://github.com/gradle/test-retry-gradle-plugin">Test Retry Gradle Plugin</a> and new <a href="https://gradle.com/blog/flaky-tests/">flaky test analysis features</a> help you eradicate flaky tests.</p>
<p>Let us know what you think here on <a href="https://twitter.com/gradle">on Twitter</a>.</p>
<script type="text/javascript">window.localStorage.setItem("preferred-sample-language", "groovy");</script>
Optional dependencies are not optional2020-01-07T00:00:00-05:00https://blog.gradle.org/optional-dependenciesCédric Champeau
<p>In a <a href="https://blog.gradle.org/addressing-logging-complexity-capabilities">previous blog post</a>, we demonstrated how <a href="https://docs.gradle.org/6.0.1/userguide/component_capabilities.html">capabilities</a> could be used to elegantly solve the problem of having multiple logging frameworks on the classpath.
In this post, we will again use this concept in a different context: <em>optional dependencies</em>.</p>
<p>At Gradle, we often say that there are no optional dependencies: there are dependencies which are required <em>if</em> you use a specific feature.
Let’s explain why.</p>
<h2 id="optional-dependencies">Optional dependencies</h2>
<p>Until recently, Gradle didn’t offer any way to publish <em>optional dependencies</em>, which is something which puzzled a number of Apache Maven™ users.
To understand in what context optional dependencies are used, let’s look at a real world project.
The <a href="https://pdfbox.apache.org">Apache PDFBox</a> library declares the following optional dependencies in its <a href="https://repo1.maven.org/maven2/org/apache/pdfbox/pdfbox/2.0.17/pdfbox-2.0.17.pom">POM file</a>:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><dependencies></span>
<span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.bouncycastle<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>bcmail-jdk15on<span class="nt"></artifactId></span>
<span class="nt"><optional></span>true<span class="nt"></optional></span>
<span class="nt"></dependency></span>
<span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.bouncycastle<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>bcprov-jdk15on<span class="nt"></artifactId></span>
<span class="nt"><optional></span>true<span class="nt"></optional></span>
<span class="nt"></dependency></span>
<span class="nt"></dependencies></span>
</code></pre></div></div>
<p>Those are 2 dependencies on a specific component, the <a href="http://www.bouncycastle.org">BouncyCastle</a> cryptography library.</p>
<p>Now let’s imagine your project depends on PDFBox, either using Gradle:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">dependencies</span> <span class="p">{</span>
<span class="nf">implementation</span><span class="p">(</span><span class="s">"org.apache.pdfbox:pdfbox:2.0.17"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>or using Apache Maven™:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><dependencies></span>
<span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.apache.pdfbox<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>pdfbox<span class="nt"></artifactId></span>
<span class="nt"><version></span>2.0.17<span class="nt"></version></span>
<span class="nt"></dependency></span>
<span class="nt"></dependencies></span>
</code></pre></div></div>
<p>Now if you look at the dependencies that both <a href="https://scans.gradle.com/s/r5h7w4zgi2os2/console-log#L10-L12">Maven</a> and <a href="https://scans.gradle.com/s/ka72ph6cu7xes/dependencies?expandAll&toggled=W1swLDAsWzBdXSxbMCwwLFswLDFdXV0">Gradle</a> resolve, you will see that the Bouncycastle library is absent.
This is because it’s defined as an <em>optional</em> dependency.
There are multiple problems with optional dependencies as they are defined today:</p>
<ul>
<li>Library authors <em>know</em> why a dependency is optional, but consumers <em>don’t</em>: how do you know when you should add <code class="language-plaintext highlighter-rouge">bcprov-jdk15on</code>?</li>
<li>Optional dependencies are mixed up: how do you know when to add <code class="language-plaintext highlighter-rouge">bcprov-jdk15on</code> and if <code class="language-plaintext highlighter-rouge">bcmail-jdk15on</code> should be added as well?</li>
<li>Optional dependencies are ignored by both Maven and Gradle when resolving transitive dependencies: the information is purely documentation that may help a user to manually add additional dependencies to a build.</li>
</ul>
<p>In our example, let’s imagine you’d like to generate <em>signed PDFs</em>.
By declaring the dependencies like above, you’ll soon realize that you are missing dependencies.
To figure out which dependencies are missing, you could <a href="https://repo1.maven.org/maven2/org/apache/pdfbox/pdfbox/2.0.17/pdfbox-2.0.17.pom">look at the POM file of PDFBox</a> and guess <em>which version of Bouncycastle</em> you need to use just by reading this.</p>
<p>Said differently, fixing is an educated guess: because you <em>kind of</em> know that Bouncycastle is related to security, you figure out that maybe if you add the dependencies, it will work.
Problem solved?</p>
<h2 id="from-optional-dependencies-to-features">From optional dependencies to features</h2>
<p>The reality is that the dependency on Bouncycastle is <em>not</em> optional: <strong>it’s required if you want to sign PDFs</strong>.
There’s an implicit feature of PDFBox which is “signing”, and if, and only if, you use that feature, then you need a couple more dependencies.
Wouldn’t it be nice if the build tools allowed library authors to express that?</p>
<p>That’s exactly what Gradle’s <a href="https://docs.gradle.org/current/userguide/feature_variants.html">feature variants</a> are for!</p>
<p>For the sake of this demonstration, let’s imagine that PDFBox would use Gradle to build their project instead of Maven.
Then they could declare a <em>feature</em> using this:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">java</span> <span class="o">{</span>
<span class="n">registerFeature</span><span class="o">(</span><span class="s1">'signing'</span><span class="o">)</span> <span class="o">{</span>
<span class="n">usingSourceSet</span><span class="o">(</span><span class="n">sourceSets</span><span class="o">.</span><span class="na">main</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This declares that PDFbox has a feature named <code class="language-plaintext highlighter-rouge">signing</code>, and that this feature is “using the main source set”.
In Gradle terms, it means that the feature is using the same source directory as the main library (<code class="language-plaintext highlighter-rouge">src/main/java</code>), combining the main sources of the library and the ones that perform signing using Bouncycastle.
Gradle also offers the ability to write a feature in separate source set (<code class="language-plaintext highlighter-rouge">src/myFeature/java</code>) to isolate the code of the feature from the main code and publish it in a separate jar.</p>
<p>Now that the <em>signing</em> feature is defined, the dependencies <em>specific to this feature</em> can be declared:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dependencies</span> <span class="o">{</span>
<span class="n">signingImplementation</span><span class="o">(</span><span class="s2">"org.bouncycastle:bcmail-jdk15on:1.64"</span><span class="o">)</span>
<span class="n">signingImplementation</span><span class="o">(</span><span class="s2">"org.bouncycastle:bcprov-jdk15on:1.64"</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>That’s all!
But what’s the benefit compared to declaring optional dependencies in Maven?</p>
<p>Well, if only we look at the POM file that Gradle would generate…</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.bouncycastle<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>bcmail-jdk15on<span class="nt"></artifactId></span>
<span class="nt"><version></span>1.64<span class="nt"></version></span>
<span class="nt"><optional></span>true<span class="nt"></optional></span>
<span class="nt"></dependency></span>
<span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.bouncycastle<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>bcprov-jdk15on<span class="nt"></artifactId></span>
<span class="nt"><version></span>1.64<span class="nt"></version></span>
<span class="nt"><optional></span>true<span class="nt"></optional></span>
<span class="nt"></dependency></span>
</code></pre></div></div>
<p>…It’s exactly the same as the one published by Maven!
Gradle offers the ability to define and publish optional dependencies by defining features and feature specific dependencies and a Maven user can consume this POM file generated by Gradle similar to the corresponding Maven POM file: look at the POM file directly and figure out what dependencies to add manually.</p>
<p>But, as a Gradle user, the benefit is much higher because things can be expressed in terms of features instead of optional dependencies.
Imagine that you need PDFBox <strong>and</strong> its signing feature.
Then you need to declare two dependencies:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">dependencies</span> <span class="p">{</span>
<span class="nf">implementation</span><span class="p">(</span><span class="s">"org.apache.pdfbox:pdfbox:2.0.17"</span><span class="p">)</span>
<span class="nf">implementation</span><span class="p">(</span><span class="s">"org.apache.pdfbox:pdfbox:2.0.17"</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">capabilities</span> <span class="p">{</span>
<span class="nf">requireCapability</span><span class="p">(</span><span class="s">"org.apache.pdfbox:pdfbox-signing"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We have two distinct dependency declarations here:</p>
<ul>
<li>The first one tells Gradle that we need PDFBox. It’s the “main dependency”.</li>
<li>The second one tells Gradle that we also want PDFBox’s <code class="language-plaintext highlighter-rouge">signing</code> feature (<code class="language-plaintext highlighter-rouge">pdfbox-signing</code>).</li>
</ul>
<p>The second dependency “points to” a different <a href="https://docs.gradle.org/6.0.1/userguide/feature_variants.html">variant</a> because Gradle, by convention, created a <a href="https://docs.gradle.org/6.0.1/userguide/component_capabilities.html">capability</a> corresponding to the feature name declared by PDFBox.</p>
<p>The big advantage is that users don’t have to figure out what dependencies they need to get signing working now: <a href="https://scans.gradle.com/s/qco3dj3ca2sry/dependencies?expandAll&toggled=W1swLDAsWzBdXSxbMCwwLFswLDFdXSxbMCwwLFszXV1d">they will get them transitively</a>!</p>
<p>It’s also interesting to notice that because we defined the dependency to Bouncycastle as an <em>implementation</em> dependency, the consumer doesn’t need it at compile time, only at runtime.
That’s why the dependency doesn’t appear on the <a href="https://scans.gradle.com/s/epw7nk3cnidjw/dependencies?expandAll">compile classpath</a>!</p>
<h2 id="what-makes-this-possible">What makes this possible?</h2>
<p>The enabler for this is again <a href="https://docs.gradle.org/current/userguide/publishing_gradle_module_metadata.html">Gradle Module Metadata</a>.
This file, just like the <code class="language-plaintext highlighter-rouge">pom.xml</code> file, contains metadata which the dependency resolution uses to find transitive dependencies.</p>
<p>In this case, the generated Gradle Module Metadata file for our “fake” PDFBox library contains the following:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"signingRuntimeElements"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"..."</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.bouncycastle"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bcmail-jdk15on"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"requires"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.64"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.bouncycastle"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bcprov-jdk15on"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"requires"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.64"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"files"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"pdfbox-2.0.17-gradle.jar"</span><span class="p">,</span><span class="w">
</span><span class="s2">"..."</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"capabilities"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.apache.pdfbox"</span><span class="p">,</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"pdfbox-signing"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2.0.17-gradle"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>It actually defines an additional variant, <code class="language-plaintext highlighter-rouge">signingRuntimeElements</code>, representing the <em>signing</em> feature we defined in the Gradle build above.
This variant includes the feature specific dependencies to Bouncycastle and declares the <code class="language-plaintext highlighter-rouge">pdfbox-signing</code> capability, which we used to select the feature in our dependency declarations.
This way, a consumer requesting this capability will properly resolve the required transitive dependencies to Bouncycastle!</p>
<h2 id="conclusion">Conclusion</h2>
<p>In this blog post, we’ve highlighted that:</p>
<ol>
<li>Optional dependencies are <strong>not optional</strong>: they are always required if you use a specific feature</li>
<li>We can properly model those features in order to group the dependencies together</li>
<li>A consumer can express dependencies on a library, including specific features</li>
<li>We can do this while maintaining compatibility with Maven</li>
</ol>
<p>It’s also worth noting that there’s no limit to the number of features a library declares.
In fact, if you use the <a href="https://docs.gradle.org/current/6.0.1/java_testing.html#sec:java_test_fixtures">Java Test Fixtures Plugin</a>, Gradle automatically declares a feature for each of the test fixtures, features that consumers <em>can</em> decide whether or not to depend on!</p>
<p>To get more information about how to use feature variants in your own builds, please head over to our <a href="https://docs.gradle.org/6.0.1/userguide/feature_variants.html">userguide</a>.</p>
Addressing the complexity of the Java logging ecosystem with capabilities2019-12-17T00:00:00-05:00https://blog.gradle.org/addressing-logging-complexity-capabilitiesLouis Jacomet
<p>Gradle 6.0 comes with a number of improvements around dependency management that we present in
<a href="https://blog.gradle.org/avoiding-dependency-hell-gradle-6">a series</a> of <a href="https://blog.gradle.org/alignment-with-gradle-module-metadata">blog posts</a>.
In this post we explore the detection of incompatible dependencies on the classpath, through the concept of <em>capabilities</em>.</p>
<p>In order to illustrate this concept, we will look at the state of logging for Java applications and libraries.
Aside from the Java core libraries providing <code class="language-plaintext highlighter-rouge">java.util.logging</code> (JUL), there are a number of logging libraries available to developers, for example:</p>
<ul>
<li><a href="https://logging.apache.org/log4j/1.2/index.html">Apache Log4J</a></li>
<li><a href="http://www.slf4j.org">Slf4J</a>, potentially combined with <a href="http://logback.qos.ch/">LOGBack</a></li>
<li><a href="http://commons.apache.org/proper/commons-logging/">Apache Commons Logging</a></li>
<li><a href="https://logging.apache.org/log4j/2.x/">Apache Log4J 2</a></li>
<li>Google’s <a href="https://github.com/google/flogger#flogger-a-fluent-logging-api-for-java">Flogger</a></li>
</ul>
<!--break-->
<h2 id="the-logging-problem">The logging problem</h2>
<p>With such a vast offering, it is no surprise that different libraries use <em>different logging APIs</em>.
Combined with the sometimes poor-quality metadata, it is quite common to end up with <em>multiple logging implementations</em> on the runtime classpath of an application.</p>
<p>This leads, amongst other things, to this well-known Slf4J warning:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:.../slf4j-log4j12-1.7.29.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:.../logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
</code></pre></div></div>
<p>While one could say “This is only a warning”, it shows us a place where problems can and do happen.
One of our users, the engineering team at Netflix, told us the following story:
By selecting the wrong Slf4J logging binding, the log files in one of their systems ended up in the wrong location – causing a disk to fill up that eventually crashed the system.
They are surely not the only ones having experienced something like this.</p>
<p>Another example: the Slf4J documentation clearly states that if you mix the <em>bridging</em> and <em>delegating</em> JARs for a given integration, <a href="http://www.slf4j.org/codes.html#jclDelegationLoop">you will encounter a <code class="language-plaintext highlighter-rouge">StackOverflowError</code></a>.</p>
<p>Log4J 2 adds its own complexity to the landscape since it also offers bridging options, that include Slf4J bridging.</p>
<p>If we take a look at the different libraries, the way they interact, and the options they offer, we end up with the following graph:</p>
<p><img src="/images/logging-capabilities/logging-landscape-base.png" alt="Java logging landscape" /></p>
<p>Figuring out the right combination of libraries based on your choice of logging framework requires studying the compatibility notes of the frameworks you use <strong>and the ones you do not use</strong>, as they can be included without you noticing via transitive dependencies.
In addition, none of these requirements are currently made available to build tools, meaning the tools cannot help developers by letting them know about invalid configurations or even pick the right combination based on a chosen logging stack.</p>
<h2 id="automatic-detection-of-invalid-logging-setups">Automatic detection of invalid logging setups</h2>
<p>With the concept of <a href="https://docs.gradle.org/6.0.1/userguide/component_capabilities.html"><em>capabilities</em></a> in Gradle, it becomes possible to provide information to the build tool so that it can, at the minimum, detect invalid setups.</p>
<p>A <strong>capability</strong> is essentially an identifier for a feature provided by a software module. Multiple modules can provide implementations of the same capability, which allows Gradle to detect if different modules are in conflict.
This can be applied directly to two different implementations of the same logging API when they are on the dependency graph.</p>
<p>There are different ways to provide capability information: directly through metadata of a component, by adding it manually to a build or by providing it through a plugin.</p>
<p>We have created such a plugin for the logging domain, which captures all the required information for the logging libraries mentioned above and allows you to resolve conflicts.
<a href="https://plugins.gradle.org/plugin/dev.jacomet.logging-capabilities">The <code class="language-plaintext highlighter-rouge">dev.jacomet.logging-capabilities</code> Gradle plugin</a> will make sure you are never surprised by an invalid logging configuration at runtime since your project will report problems at build time!</p>
<p>Let’s review some of the situations it handles.</p>
<h3 id="slf4j-and-its-multiple-bindings">Slf4J and its multiple bindings</h3>
<p>Since Slf4J warns that multiple bindings are problematic, it effectively creates an <em>exclusive</em> implementation relationship. Any module that implements the <code class="language-plaintext highlighter-rouge">slf4j-api</code> to provide a binding <em>cannot</em> live on the classpath with another such implementation.</p>
<p>In order to detect that we have multiple Slf4J bindings in a given dependency graph, the plugin adds the capability <code class="language-plaintext highlighter-rouge">dev.jacomet.logging:slf4j-impl:1.0</code> to all of the following modules: <code class="language-plaintext highlighter-rouge">logback-classic</code>, <code class="language-plaintext highlighter-rouge">slf4j-simple</code>, <code class="language-plaintext highlighter-rouge">slf4j-log4j12</code>, <code class="language-plaintext highlighter-rouge">slf4j-jdk14</code>, <code class="language-plaintext highlighter-rouge">log4j-slf4j-impl</code> and <code class="language-plaintext highlighter-rouge">slf4j-jcl</code>.</p>
<p>With that information in place, it becomes illegal to have two Slf4J bindings in any resolved dependency graph, enforcing at <em>build time</em> what was only reported at <em>runtime</em> by Slf4J before.</p>
<p>Here is an example build output with such a failure:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':doIt'.
> Could not resolve all files for configuration ':runtimeClasspath'.
> Could not resolve org.slf4j:slf4j-simple:1.7.27.
Required by:
project :
> Module 'org.slf4j:slf4j-simple' has been rejected:
Cannot select module with conflict on capability 'dev.jacomet.logging:slf4j-impl:1.0' also provided by [ch.qos.logback:logback-classic:1.2.3(runtime)]
> Could not resolve ch.qos.logback:logback-classic:1.2.3.
Required by:
project :
> Module 'ch.qos.logback:logback-classic' has been rejected:
Cannot select module with conflict on capability 'dev.jacomet.logging:slf4j-impl:1.0' also provided by [org.slf4j:slf4j-simple:1.7.27(runtime)]
</code></pre></div></div>
<h3 id="log4j-or-a-different-logging-solution">Log4J or a different logging solution?</h3>
<p>Log4J had its first release in 2001, before JUL even existed, and has not had a release since <code class="language-plaintext highlighter-rouge">1.2.17</code> in 2012.</p>
<p>Given this, as developers, you will probably prefer a more recent solution, like an Slf4J binding or Log4J 2, to act as your logging framework but you may be using transitive dependencies that were developed against the Log4J API.</p>
<p>In order to apply your choice, you need to be aware of a few potential conflicts:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">log4j:log4j</code> needs to be replaced by either <code class="language-plaintext highlighter-rouge">log4j-over-slf4j</code> from Slf4J or <code class="language-plaintext highlighter-rouge">log4j-1.2-api</code> from Log4J 2</li>
<li>These two replacements are themselves exclusive</li>
<li>If you use <code class="language-plaintext highlighter-rouge">log4j-over-slf4j</code> you cannot use <code class="language-plaintext highlighter-rouge">slfj-log4j12</code></li>
</ul>
<p>Once again the <code class="language-plaintext highlighter-rouge">dev.jacomet.logging-capabilities</code> plugin takes care of declaring the necessary capabilities for you:</p>
<ul>
<li>It will add the <code class="language-plaintext highlighter-rouge">dev.jacomet.logging:slf4j-vs-log4j</code> capability to <code class="language-plaintext highlighter-rouge">log4j-over-slf4j</code> and <code class="language-plaintext highlighter-rouge">slfj-log4j12</code></li>
<li>It will add the <code class="language-plaintext highlighter-rouge">dev.jacomet.logging:slf4j-vs-log4j2-log4j</code> capability to <code class="language-plaintext highlighter-rouge">log4j:log4j</code>, <code class="language-plaintext highlighter-rouge">log4j-over-slf4j</code> and <code class="language-plaintext highlighter-rouge">log4j-1.2-api</code></li>
</ul>
<p>This gives you the guarantee that you will not mix incompatible bridging and implementations of Log4J in any resolved dependency graph.</p>
<h3 id="a-comprehensive-solution">A comprehensive solution</h3>
<p><img src="/images/logging-capabilities/logging-landscape-conflicts.png" alt="Java logging landscape and conflicts" /></p>
<p>As you can see on the enhanced graph, there are many other problematic module combinations:</p>
<ul>
<li>Similarly to the Log4J replacement, JUL can be replaced by either Slf4J or Log4J 2</li>
<li>For Apache Commons Logging, the Log4J2 integration <em>requires</em> <code class="language-plaintext highlighter-rouge">commons-logging</code> while the Slf4J one <em>replaces</em> it</li>
<li>…</li>
</ul>
<p>For all these potential conflicts, the plugin <code class="language-plaintext highlighter-rouge">dev.jacomet.logging-capabilities</code> registers the necessary capabilities to detect all invalid combinations. Head to the <a href="https://github.com/ljacomet/logging-capabilities#detection-of-invalid-logging-configurations">plugin documentation</a> for a comprehensive list of the capabilities and their role.</p>
<h3 id="behind-the-scenes">Behind the scenes</h3>
<p>The plugin leverages Gradle <a href="https://docs.gradle.org/6.0.1/userguide/component_metadata_rules.html">component metadata rules</a> to add the capability information.</p>
<p><a href="https://github.com/ljacomet/logging-capabilities/blob/b5ce6bc1ff2a143b4fc8bc62af67d40231691b60/src/main/java/dev/jacomet/gradle/plugins/logging/LoggingCapabilitiesPlugin.java#L80-L89">Head over to the plugin code</a> to see how it adds a capability <code class="language-plaintext highlighter-rouge">dev.jacomet.logging:slf4j-impl:1.0</code> to all the modules configured with the rule.</p>
<p>Similarly, rules are added for all the possible conflicts that the graph above identifies.</p>
<h2 id="enhancing-the-logging-ecosystem-at-publication-time">Enhancing the logging ecosystem at publication time</h2>
<p>With Gradle Module Metadata, the concepts presented in the previous section could be applied to the <em>published metadata</em> of logging libraries.
It would make the use of custom <code class="language-plaintext highlighter-rouge">ComponentMetadataRule</code>s or a plugin like the one above obsolete, because the information would be encoded by the library author into the library’s <em>published</em> metadata.</p>
<p>As a library author, capabilities can be added for publication as shown in <a href="https://docs.gradle.org/6.0.1/userguide/component_capabilities.html#declaring_additional_capabilities_for_a_local_component">the Gradle documentation</a>.</p>
<h3 id="identifying-which-capability-to-declare">Identifying which capability to declare</h3>
<p>An important part of the work is to determine the coordinates of these shared capabilities.
Ideally that choice would be made by the original library which offers an extensible system.
Then, third-party implementers would be able to conform to the capability declaration in their implementation.</p>
<h2 id="i-dont-want-my-build-to-break-i-want-gradle-to-fix-it">I don’t want my build to break, I want Gradle to fix it!</h2>
<p>We have seen how capabilities are used to detect conflicts and fail a build in case of such a conflict.
But that alone does not help us if we can not fix the detected conflicts.
For this, Gradle offers capability resolution strategies.</p>
<p>The <code class="language-plaintext highlighter-rouge">dev.jacomet.logging-capabilities</code> plugin already sets up such resolution strategies and offers simple constructs to select and activate them.
You can declaratively express your logging choices and the plugin makes sure to enhance your build with the relevant capabilities resolution and substitution rules so that only necessary logging libraries appear on the classpath.</p>
<p>The following will make sure Log4J 2 is used as the logger implementation:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>plugins {
`java-library`
id("dev.jacomet.logging-capabilities")
}
loggingCapabilities {
enforceLog4J2()
}
</code></pre></div></div>
<p>It will:</p>
<ul>
<li>Configure Log4J 2 to bridge Slf4J, if there are Slf4J bridges in the graph</li>
<li>Configure the bridging of JUL with Log4J 2</li>
<li>Configure the bridging of <code class="language-plaintext highlighter-rouge">commons-logging</code> only if required</li>
<li>Replace <code class="language-plaintext highlighter-rouge">log4j</code> with <code class="language-plaintext highlighter-rouge">log4j-1.2-api</code> only if required</li>
</ul>
<p>However it will not <em>add</em> Log4J 2 dependencies, which are to be added as dependencies – directly or transitively.</p>
<p>For a complete overview of the choice that can be expressed, head over to <a href="https://github.com/ljacomet/logging-capabilities#expressing-preference-over-a-logging-solution">the plugin documentation</a>.</p>
<h3 id="behind-the-scenes-1">Behind the scenes</h3>
<p>Gradle offers <a href="https://docs.gradle.org/current/userguide/dependency_capability_conflict.html#sub:selecting-between-candidates">an API</a> to indicate how to resolve capabilities conflicts.</p>
<p><a href="https://github.com/ljacomet/logging-capabilities/blob/b5ce6bc1ff2a143b4fc8bc62af67d40231691b60/src/main/java/dev/jacomet/gradle/plugins/logging/extension/LoggingCapabilitiesExtension.java#L315-L318">The plugins uses it</a> to tell Gradle that in case of a conflict on <code class="language-plaintext highlighter-rouge">dev.jacomet.logging:slf4j-impl</code>, the engine must select the module <code class="language-plaintext highlighter-rouge">org.slf4j:slf4j-simple:1.7.25</code> for the test runtime classpath.
The conflict resolution logic inspects the available candidates and performs a conditional selection.</p>
<p>Note that in the example above, another Slf4J implementation could be used if there was no capability conflict.
It is however quite likely that you have the <code class="language-plaintext highlighter-rouge">slf4j-simple</code> dependency declared in your build if you intend to use it.
And the plugin requires it in order to work properly.</p>
<h2 id="conclusion">Conclusion</h2>
<p>We have seen that capabilities are a modelling concept provided by Gradle to express mutual exclusivity between different libraries.
As shown on the logging use case, they enable Gradle’s dependency resolution to fail when conflicting implementations of a feature are found in the dependency graph.
<a href="alignment-with-gradle-module-metadata#conclusion">Similarly to alignment</a>, using capabilities with Gradle Module Metadata gives library authors the power to share more knowledge about when and in which combinations their library is intended to be used.
With this information available, Gradle offers APIs for build authors to resolve conflicts in their own build <em>declaratively</em> without hackery.</p>
<p>The use cases that can be addressed with the <em>capabilities</em> concept go beyond the logging use case demonstrated here.
Libraries that have changed coordinates, that exist in multiple formats (like <code class="language-plaintext highlighter-rouge">cglib</code> and <code class="language-plaintext highlighter-rouge">cglib-nodep</code>), or simply have different feature sets, can all leverage this concept to express that the presence of more than one module on a classpath should be considered an error.</p>
What’s new in Gradle 6.02019-11-26T00:00:00-05:00https://blog.gradle.org/what-is-new-in-gradle-6Eric Wendelin
<p>Gradle 6.0 is the culmination of several years of innovative improvements in Dependency Management. Embracing the idea that there is more to software composition and reuse than just putting a set of jar files on the classpath, Gradle now offers a new metadata format to richly define software components that are often made up of multiple files, different variants and specific constraints on their dependencies.</p>
<p>In particular, in this webcast Developer Advocate Jenn Strater and Gradle Engineer Jendrik Johannes discuss:</p>
<ol>
<li>What’s New in Dependency Management</li>
<li>Java, Groovy, and Scala toolchain improvements</li>
<li>New features for plugin authors</li>
</ol>
<p style="text-align:center;"><iframe width="560" height="315" style="text-align:center;" src="https://www.youtube.com/embed/nHXy2due_fA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe></p>
<p>You can access the slides of the webinar <a href="https://gradle.github.io/webinar-gradle-6/">here</a>. If you want to try things out, feel free to use the <a href="https://github.com/gradle/webinar-gradle-6/tree/master/demos/hello-gradle-6">demo project</a> that we used during the webinar.</p>
<p>Thanks to everyone that attended the webinar and thanks for your questions!</p>
Automatically align Dependencies with Platforms and Gradle Module Metadata2019-11-25T00:00:00-05:00https://blog.gradle.org/alignment-with-gradle-module-metadataJendrik Johannes
<p><em>This blogpost has been updated to reflect that Jackson started to published Gradle Module Metadata with version 2.12.0 using the recommendations from this post.</em></p>
<p>In the <a href="https://blog.gradle.org/avoiding-dependency-hell-gradle-6">previous post</a> about dependency management with Gradle 6, we saw that growing builds can quickly end up in dependency hell.
Unexpected results become particularly hard to analyze if they are introduced at the bottom of the dependency graph and propagate up through transitive dependencies.
It is unfortunate that some of these issues could be avoided if the authors of libraries, which form the bottom of the dependency graph, had the means to express all the knowledge about the versioning of said libraries in the libraries’ metadata.</p>
<p>A typical example of such a library is the widely used JVM utility library <a href="https://github.com/FasterXML/jackson">Jackson</a>.
If several components of the library are part of the dependency graph, alignment of versions can be an issue.</p>
<p>The Jackson library is made up of only three core modules: <code class="language-plaintext highlighter-rouge">jackson-annotations</code>, <code class="language-plaintext highlighter-rouge">jackson-core</code> and <code class="language-plaintext highlighter-rouge">jackson-databind</code>.
Still, it is easily possible to end up in a situation where a higher version is selected for <code class="language-plaintext highlighter-rouge">jackson-core</code> than for <code class="language-plaintext highlighter-rouge">jackson-databind</code>.
This situation is illustrated in the following example, where the two modules end up having different versions: <code class="language-plaintext highlighter-rouge">jackson-core:2.9.2</code> and <code class="language-plaintext highlighter-rouge">jackson-databind:2.8.9</code>.</p>
<p><img src="/images/gradle-6-dm/jackson-dependency-graph.png" alt="Transitive dependencies resolve to jackson-core:2.9.2 but jackson-databind:2.2.2" /></p>
<p>In the example, which you can explore as a <a href="https://scans.gradle.com/s/yunliedpspfte/dependencies?toggled=W1swXSxbMCwwXSxbMCwwLFsxMzddXSxbMCwwLFsxMzcsMTQ0XV1d">build scan</a>,
one transitive dependency (<em>tika-parsers</em>) upgraded <code class="language-plaintext highlighter-rouge">jackson-core</code> to <code class="language-plaintext highlighter-rouge">2.9.2</code> as Gradle resolves the conflict between <code class="language-plaintext highlighter-rouge">2.8.9</code> and <code class="language-plaintext highlighter-rouge">2.9.2</code> to the higher version.
<code class="language-plaintext highlighter-rouge">jackson-databind</code> however, added via another dependency (<em>keycloak-core</em>), is kept on <code class="language-plaintext highlighter-rouge">2.8.9</code> as the information that it should be aligned, i.e. upgraded together, with <code class="language-plaintext highlighter-rouge">jackson-core</code> is missing.</p>
<h2 id="boms-are-great-but-we-dont-use-them-enough">BOMs are great, but we don’t use them (enough)</h2>
<p>The situation we face in this scenario is that the Jackson modules’ versions should be aligned, but that information is not published due to limitations of the pom metadata format.
What Jackson is publishing is a BOM (bill of materials), a <code class="language-plaintext highlighter-rouge">pom.xml</code> containing only dependency version information.
The BOM contains some alignment information as this excerpt from <a href="https://repo1.maven.org/maven2/com/fasterxml/jackson/jackson-bom/2.9.2/jackson-bom-2.9.2.pom">jackson-bom-2.9.2.pom</a> shows:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><dependencyManagement>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.9.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.2</version>
</dependency>
...
</dependencyManagement>
...
</code></pre></div></div>
<p>However, to use this alignment information, both Maven and Gradle users need to <strong>explicitly</strong> depend on the BOM in their own build.</p>
<p><em>Maven build importing the jackson-bom:</em></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><dependencyManagement>
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson</groupId>
<artifactId>jackson-bom</artifactId>
<version>2.8.9</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
</code></pre></div></div>
<p><em>Gradle 5.x build with platform dependency on the jackson-bom:</em></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dependencies {
// depend on a platform and enforce all version entries (Maven semantics)
implementation(enforcedPlatform("com.fasterxml.jackson:jackson-bom:2.8.9"))
// depend on a platform and do dependency conflict resolution with all entries
implementation(platform("com.fasterxml.jackson:jackson-bom:2.8.9"))
}
</code></pre></div></div>
<p>There are several issues with this approach: knowing a BOM for Jackson exists, deciding which version of the Jackson BOM to use, and updating that version when the build evolves.</p>
<p>If we use Maven, the versions provided in the BOM are enforced.
This means that a request for a higher version by any of the dependencies would be silently ignored.
In the example above, if we chose the <code class="language-plaintext highlighter-rouge">2.8.9</code> BOM, <code class="language-plaintext highlighter-rouge">jackson-core</code> would be downgraded to <code class="language-plaintext highlighter-rouge">2.8.9</code>.
This can cause issues, as <code class="language-plaintext highlighter-rouge">tika-parsers</code> requires <code class="language-plaintext highlighter-rouge">jackson-core:2.9.2</code> and might break with the downgrade.
Consequently, the build author has to carefully choose the BOM’s version and revisit that choice when dependencies change.
Without tool support, this can become unmanageable if multiple BOMs are used.</p>
<h2 id="the-missing-link-platform-dependencies-on-boms">The missing link: Platform dependencies on BOMs</h2>
<p>Gradle 5.0 introduced the ability to declare a <em>platform dependency</em> to a BOM.
In this case, versions are not silently enforced, but the entries in the BOM participate in conflict resolution.
In the example, however, this means that we are back to the initial problem:
We select the <code class="language-plaintext highlighter-rouge">2.8.9</code> BOM in the build script, which recommends version <code class="language-plaintext highlighter-rouge">2.8.9</code> for <code class="language-plaintext highlighter-rouge">jackson-core</code>, but <code class="language-plaintext highlighter-rouge">jackson-core</code> is upgraded to <code class="language-plaintext highlighter-rouge">2.9.2</code> by a transitive dependency.</p>
<blockquote>
<p><strong>What we are missing is an automatic upgrade of the platform (<code class="language-plaintext highlighter-rouge">jackson-bom</code>) to the highest selected version (<code class="language-plaintext highlighter-rouge">2.9.2</code>) as well.
This cannot be expressed in pom metadata, but it can with Gradle 6 using Gradle Module Metadata.</strong></p>
</blockquote>
<h2 id="the-gradle-module-metadata-answer">The Gradle Module Metadata answer</h2>
<p>With <a href="gradle-metadata-1.0">Gradle Module Metadata</a>, the Jackson team now publishes platform dependencies for each version of <code class="language-plaintext highlighter-rouge">jackson-core</code> with the information about which version of the platform (<code class="language-plaintext highlighter-rouge">jackson-bom</code>) it belongs to.
For example, if <code class="language-plaintext highlighter-rouge">jackson-core</code> would be built with Gradle, this platform dependency could be added to its build script:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dependencies {
// I belong to the 'jackson-bom' platform with the same version
api(platform("com.fasterxml.jackson:jackson-bom:${project.version}"))
...
}
</code></pre></div></div>
<p>If using Gradle 6, Gradle Module Metadata is published by default and thus includes the platform dependency.
In a similar fashion, the platform dependency can be added to <code class="language-plaintext highlighter-rouge">jackson-databind</code> and <code class="language-plaintext highlighter-rouge">jackson-annotations</code>.
With this additional information, the dependency graph from the beginning of this post will look like this:</p>
<p><img src="/images/gradle-6-dm/jackson-dependency-graph-with-bom.png" alt="Transitive dependencies resolve to jackson-core:2.9.2 but jackson-databind:2.8.9" /></p>
<p>The updated graph, which you can also explore in this <a href="https://scans.gradle.com/s/vh57xjlphhxjs/dependencies?toggled=W1swXSxbMCwwXSxbMCwwLFsxNDVdXSxbMCwwLFsxNDUsMTUyXV0sWzAsMCxbMTQ1LDE1MV1dXQ">build scan</a>, shows two things:
First, there is a new node, <code class="language-plaintext highlighter-rouge">jackson-bom</code>.
The edges <strong>towards</strong> the <code class="language-plaintext highlighter-rouge">jackson-bom</code> come from the Jackson modules.
These originate from the published platform dependencies and thus the <code class="language-plaintext highlighter-rouge">jackson-bom</code> is automatically added without explicitly depending on it in the build.
Second, each module version brings in the <code class="language-plaintext highlighter-rouge">jackson-bom</code> that fits its version – <code class="language-plaintext highlighter-rouge">jackson-databind:2.8.9</code> brings in <code class="language-plaintext highlighter-rouge">jackson-bom:2.8.9</code>; <code class="language-plaintext highlighter-rouge">jackson-core:2.9.2</code> brings in <code class="language-plaintext highlighter-rouge">jackson-bom:2.9.2</code>.
Gradle then resolves the version conflict of jackson-bom to the higher version, which in turn adds the dependency constraints of all the higher module versions in the graph, eventually causing all components to align on the highest version.</p>
<blockquote>
<p><strong>With Gradle Module Metadata, platform dependencies are published for an automatic updated of the platform (<code class="language-plaintext highlighter-rouge">jackson-bom</code>) to the highest selected version.</strong></p>
</blockquote>
<h2 id="adding-the-missing-bits-to-existing-jackson-metadata">Adding the missing bits to existing Jackson metadata</h2>
<blockquote>
<p><strong>Starting with the Jackson 2.12.0 release, Jackson published Gradle Module Metadata as proposed in this blogpost. The following is not needed to align to 2.12.0 or higher.</strong></p>
</blockquote>
<p>Since Gradle Module Metadata is a new format, adoption of it will take time.
To bridge this gap, Gradle 6 allows you to write <a href="https://docs.gradle.org/6.0.1/userguide/component_metadata_rules.html">component metadata rules</a> to enrich published pom metadata with the missing information when it is processed by Gradle.
For the Jackson example, you would add the following to your build script:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>open class JacksonAlignmentRule: ComponentMetadataRule {
@Inject open fun getObjects(): ObjectFactory = throw UnsupportedOperationException()
override fun execute(ctx: ComponentMetadataContext) {
if (ctx.details.id.group == "com.fasterxml.jackson.core") {
ctx.details.allVariants {
withDependencies {
add("com.fasterxml.jackson:jackson-bom:${ctx.details.id.version}") {
attributes {
attribute(Category.CATEGORY_ATTRIBUTE,
getObjects().named(Category.REGULAR_PLATFORM))
}
}
}
}
}
}
}
dependencies {
// apply the JacksonAlignmentRule rule defined above
components.all<JacksonAlignmentRule>()
}
</code></pre></div></div>
<h2 id="publishing-your-library-with-alignment">Publishing your library with alignment</h2>
<p>Examples of libraries that use the approach described in this blog post are
<a href="https://junit.org/junit5">JUnit 5</a> and Jackson.
If you are a library author, you can do the same!</p>
<p>If you use <strong>Gradle 6+</strong>, you can add alignment for the modules of your library by doing what JUnit does:</p>
<ul>
<li>Have a <a href="https://github.com/junit-team/junit5/blob/b7eddfc1d953282c2e2cf428a425e047a96b6ebf/junit-bom/junit-bom.gradle.kts#L2">java-platform project</a></li>
<li>Define <a href="https://github.com/junit-team/junit5/blob/b7eddfc1d953282c2e2cf428a425e047a96b6ebf/junit-jupiter-api/junit-jupiter-api.gradle.kts#L11">platform dependencies to that project</a></li>
<li>Use Gradle’s <a href="https://github.com/junit-team/junit5/blob/b7eddfc1d953282c2e2cf428a425e047a96b6ebf/buildSrc/src/main/kotlin/publishing-conventions.gradle.kts#L4">maven-publish plugin</a> to publish metadata with platform dependencies</li>
</ul>
<p>If you use <strong>Maven</strong>, you can make use of this <a href="https://github.com/jjohannes/gradle-module-metadata-maven-plugin">Maven plugin</a> to add alignment for the modules of your library by doing what Jackson does:</p>
<ul>
<li>Have a <a href="https://github.com/FasterXML/jackson-bom">BOM project</a></li>
<li>Configure the above mentioned plugin to <a href="https://github.com/FasterXML/jackson-bom/blob/1588acbf625560701965c3cf3ec7c8ef02c345ad/base/pom.xml#L245-L265">use your BOM for alignment</a> in a parent POM</li>
<li>Activate the plugin in the <a href="https://github.com/FasterXML/jackson-core/blob/9a77a0637f3495c8b40908dcb0bf31e56ca8bf8f/pom.xml#L127-L130">POMs of all your modules</a></li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>The Gradle Module Metadata format can be used by library authors to publish platform dependencies to align the versions of modules in their libraries.
In builds that use such libraries, Gradle performs the alignment automatically.
If you use a library that is not (yet) publishing this information, you can write your own rules to add the missing bits to the library’s metadata.</p>
<p>Version alignment is only one of many use cases that are solved with the help of Gradle Module Metadata.
In the next blog post about Gradle 6 dependency management, we will explore how dependency conflicts between different modules, and variants of modules, can be detected and resolved.
If you like to dive deeper right now, explore Gradle’s user manual sections about <a href="https://docs.gradle.org/6.0.1/userguide/core_dependency_management.html">dependency management</a>.</p>
Avoiding dependency hell with Gradle 62019-11-11T00:00:00-05:00https://blog.gradle.org/avoiding-dependency-hell-gradle-6Cédric Champeau
<p>Dependency hell is a big problem for many teams. The larger the project and its dependency graph, the harder it is to maintain it.
The solutions provided by existing dependency management tools are insufficient to effectively deal with this issue.</p>
<p>Gradle 6 aims at offering actionable tools that will help deal with these kind of problems, making dependency management more maintainable and reliable.</p>
<p>Take, for example, this anonymized dependency graph from a real world project:</p>
<p><img src="/images/gradle-6-dm/large-dependency-graph.png" alt="A large dependency graph" /></p>
<p>There are <em>hundreds</em> of different libraries in this graph.
Some are internal libraries, some are OSS libraries.
A proportion of those modules see several releases a week.
In practice, with a graph of this size, there’s no way you can avoid typical problems like:</p>
<ul>
<li><a href="https://issues.apache.org/jira/browse/HIVE-7387">2 components depending on the same module but with different, incompatible, APIs</a></li>
</ul>
<p><img src="/images/gradle-6-dm/no-such-method.png" alt="Method not found at runtime" /></p>
<ul>
<li>multiple libraries providing the same feature (<a href="https://www.baeldung.com/slf4j-classpath-multiple-bindings">a single logger API, but you end up with multiple implementations</a>)</li>
</ul>
<p><img src="/images/gradle-6-dm/slf4j-bindings.png" alt="Multiple SLF4J bindings" /></p>
<ul>
<li>and more like:
<ul>
<li>dealing with incompatible versions of a runtime (e.g: <a href="https://stackoverflow.com/questions/43568116/is-there-a-way-to-stop-scala-2-12-breaking-the-jackson-object-mapper">Scala 2.11 vs Scala 2.12</a>)</li>
<li>misaligned dependencies of a component (e.g: <a href="https://stackoverflow.com/questions/18429468/correct-set-of-dependencies-for-using-jackson-mapper">Jackson Databind 2.9.0 with Jackson Core 2.9.4</a>)</li>
<li>builds suddenly failing because of <a href="https://blog.danlew.net/2015/09/09/dont-use-dynamic-versions-for-your-dependencies/">a dynamic version upgrade</a> (version “1.+”)</li>
<li><a href="https://stackoverflow.com/questions/57980603/update-transitive-dependency">rejecting vulnerable transitive dependencies</a></li>
<li><a href="https://stackoverflow.com/questions/1517611/is-there-a-simple-way-to-remove-unused-dependencies-from-a-maven-pom-xml">removing unused dependencies</a></li>
<li>inconsistent versions between <a href="https://stackoverflow.com/questions/31841259/how-do-i-share-dependencies-between-android-modules">subprojects in the same repository</a></li>
</ul>
</li>
</ul>
<p>Dependency issues can cause many problems when building and testing your products and it can be extremely challenging to figure out, on a daily basis, what caused a regression, why the project suddenly doesn’t build anymore or what dependency is responsible for an upgrade of another dependency.</p>
<p>If you are lucky, you would get a <em>compile time error</em>, but it’s common to only see problems occurring when executing tests or even at production runtime.
In all these cases the error is often hard to trace back to the source, as it appears after the dependency resolution in the build tool has been successful.
So from a dependency management perspective everything is <em>correct</em>, while in fact it is not.</p>
<p>The reason for this mismatch is that the engine resolving dependencies does not have enough information to detect - and if possibly automatically fix - a problem.
To provide more information to the engine, modules need to carry more metadata.
The good news is that this is the focus of Gradle 6!</p>
<h2 id="introducing-gradle-6-dependency-management">Introducing Gradle 6 dependency management</h2>
<p>Gradle 6 takes a step forward and is the enabler of a new era in dependency management.
With the help of <a href="https://blog.gradle.org/gradle-metadata-1.0">Gradle Module Metadata</a>, Gradle now supports a richer, smarter dependency declaration model which enables the build tool to take better decisions, make builds more reliable, and reduce the cost of maintaining dependency graphs.</p>
<p>A lot of the problems seen in dependency management are usually the consequence of a disagreement between a <em>consumer</em> (e.g, the application you build) and a <em>producer</em> (e.g the library/dependency you use), because there isn’t enough information for the dependency management engine to make good decisions.</p>
<p>It is critical that library (e.g., Guava) or framework authors (e.g, Spring Boot, or internal framework) can express requirements in a richer way so that their users face less dependency management issues.
They should be able to express things like “use this version if you don’t know which one to use”, or “if you use this feature, then you also need those additional dependencies”.
Those are some of the many options Gradle 6 offers.</p>
<p>A typical dependency declaration is expressed in terms of <code class="language-plaintext highlighter-rouge">group</code>, <code class="language-plaintext highlighter-rouge">artifact</code> and <code class="language-plaintext highlighter-rouge">version</code> (also known as GAV coordinates, e.g. <code class="language-plaintext highlighter-rouge">com.google.guava:guava:25.1</code>).
Let’s focus for a second on the <code class="language-plaintext highlighter-rouge">version</code> part.
What does it <em>mean</em>, if you see <code class="language-plaintext highlighter-rouge">25.1</code>:</p>
<ul>
<li>was it the <em>latest release</em> at the moment you wrote the code?</li>
<li>was it one version you copied and pasted from StackOverflow and <em>it worked</em>?</li>
<li>would it work with <code class="language-plaintext highlighter-rouge">25.0</code>?</li>
<li>is it ok to upgrade to <code class="language-plaintext highlighter-rouge">26.0</code>?</li>
</ul>
<p>A direct consequence of the lack of semantics associated with a single version declaration is that we’re likely to perform <em>optimistic upgrading</em>.
We assume that because it works with <code class="language-plaintext highlighter-rouge">25.1</code>, it <em>should be fine</em> to upgrade to <code class="language-plaintext highlighter-rouge">26.0</code>.
In practice, this works pretty well, and this has been the strategy used by Gradle for years.</p>
<p>However, there are cases where optimistic upgrading breaks:</p>
<ul>
<li>major version upgrades (breaks binary compatibility)</li>
<li>vulnerabilities (you should really not include <code class="language-plaintext highlighter-rouge">1.6</code> because it has a CVE)</li>
<li>regressions (there’s a bug in <code class="language-plaintext highlighter-rouge">1.6</code>)</li>
<li>library belongs to a larger set of modules which need to share the same version (e.g Jackson Core, Databind, Annotations, …)</li>
<li>…</li>
</ul>
<p>As an example, Gradle 6 offers you the ability to express things in a richer model:</p>
<ul>
<li>you need this dependency strictly within the <code class="language-plaintext highlighter-rouge">[1.0, 2.0[</code> range (because it follows semantic versioning)</li>
<li>and <em>within the range</em>, you <em>prefer</em> <code class="language-plaintext highlighter-rouge">1.5</code> (because that’s what you have tested)</li>
<li>and you reject <code class="language-plaintext highlighter-rouge">1.6</code>, because you know it has a bug that directly affects you</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dependencies {
implementation("org.sample:sample") {
version {
strictly("[1.0, 2.0[")
prefer("1.5")
reject("1.6")
}
}
}
</code></pre></div></div>
<p>This means that if nobody else cares, the engine would choose <code class="language-plaintext highlighter-rouge">1.5</code>.
If another dependency requires <code class="language-plaintext highlighter-rouge">1.7</code>, we know we can safely upgrade to <code class="language-plaintext highlighter-rouge">1.7</code>.
However, if another dependency requires <code class="language-plaintext highlighter-rouge">2.1</code>, we can now <em>fail the build</em> because two modules disagree.</p>
<p>In addition, there is information that a producer would not know about a dependency, because it changes after a library is published: discovered bugs, vulnerabilities, incorrect transitive dependencies, etc…
This is information which can be pushed at any time to the dependency management engine as additional input!</p>
<p>It’s worth noting that the improvements Gradle provides are not only for consumers. As a library author, you have more flexibility than ever in the way you express what you produce: different modules which should have their versions aligned, libraries with optional features, a platform of suggestions for dependency versions, different binaries for different versions of a runtime and much more!</p>
<p>Gradle has been offering these features for several versions now. However their usage was mostly limited to multi-project setups. With Gradle 6, all these tools are now available to library authors and consumers alike by supporting them in published modules with Gradle Module Metadata. It enables a clearer expression of requirements and allows the engine to compute the best solution.</p>
<h2 id="gradle-module-metadata">Gradle Module Metadata</h2>
<p>Because the Gradle dependency model is richer than what other build tools offer (Ant+Ivy, Maven, Bazel …), we needed a metadata format to enable all those features for libraries published on binary repositories like Maven Central, Artifactory or Nexus.
This metadata format is basically a serialization of the Gradle model.
You can learn more about this in our <a href="https://blog.gradle.org/gradle-metadata-1.0">dedicated blog post</a>.</p>
<p>In Gradle 6.0, publication of Gradle Module Metadata is enabled by default.</p>
<p>As a library author, you shouldn’t be worried about using Gradle specific features: in all cases, publication of Maven or Ivy metadata is still possible and we did our best to map Gradle specific concepts to those formats when possible. In case it wasn’t possible, it just means some features will only be available for Gradle users, but typically Maven users wouldn’t lose anything compared to what they have today.</p>
<h2 id="in-practice">In practice</h2>
<p>Last but not least, for Gradle 6 we have significantly rewritten the <a href="https://docs.gradle.org/6.0/userguide/dependency_management.html">dependency management documentation</a> section of our userguide to make it more use case centric.</p>
<p>In the upcoming weeks, we are going to publish a number of blog posts covering different use cases in more details. In particular we’ll explain what you can do with Gradle 6, including:</p>
<ul>
<li>Declaring <a href="https://docs.gradle.org/6.0/userguide/rich_versions.html">rich version constraints</a> to express intent more clearly and let the engine find the best solutions</li>
<li>Performing centralized version declaration with <a href="https://docs.gradle.org/6.0/userguide/java_platform_plugin.html">platforms</a>.</li>
<li>Fixing issues of incompatible module versions, also known as <a href="https://docs.gradle.org/6.0/userguide/dependency_version_alignment.html">dependency version alignement</a>.</li>
<li>Getting rid of the infamous multiple logger implementations with <a href="https://docs.gradle.org/6.0/userguide/component_capabilities.html">capabilities</a>.</li>
<li>Building and consuming libraries with <a href="https://docs.gradle.org/6.0/userguide/feature_variants.html">optional features</a></li>
<li>Ensuring reproducible builds using dynamic versions with <a href="https://docs.gradle.org/6.0/userguide/dependency_locking.html">dependency locking</a></li>
<li>The different types of Java components: <a href="https://docs.gradle.org/6.0/userguide/java_library_plugin.html">libraries</a>, <a href="https://docs.gradle.org/6.0/userguide/application_plugin.html">applications</a> and <a href="https://docs.gradle.org/6.0/userguide/java_platform_plugin.html">platforms</a></li>
</ul>
<p>Gradle 6 is a major step towards better dependency management, but development doesn’t stop there: we know we still have a lot of work to do, and we’ll address your feedback, don’t hesitate!</p>
Decommissioning HTTP for Gradle Services2019-10-17T00:00:00-04:00https://blog.gradle.org/decommissioning-httpJonathan Leitschuh
<p>Starting in January 2020, <strong>Gradle services will only serve requests made with HTTPS</strong>. From that point on,
all requests made with HTTP will be denied and any builds and artifact mirrors that use a Gradle URL with the non-secure
HTTP protocol will fail.</p>
<p>If you are proxying our services through your own artifact servers like Artifactory or Nexus, you will need
to ensure that you update your mirror configurations so they are using HTTPS instead of HTTP.</p>
<h2 id="gradle-services">Gradle Services</h2>
<p>This change will impact the following services.</p>
<h3 id="plugin-portal">Plugin Portal</h3>
<p>By default, the Gradle build tool uses HTTPS when resolving plugins from the Plugin Portal.
You should be unaffected if you do not <a href="https://docs.gradle.org/current/userguide/plugins.html#sec:plugin_management">declare a custom plugin repository</a>.</p>
<p>If your organization <a href="https://plugins.gradle.org/docs/mirroring">mirrors the Plugin Portal</a>
from URL <code class="language-plaintext highlighter-rouge">plugins.gradle.org/m2/*</code>, you should check that your mirror is using HTTPS.</p>
<h3 id="gradle-distributions">Gradle Distributions</h3>
<p>Since Gradle 1.2, the Gradle wrapper has used HTTPS to download Gradle distributions. You should be unaffected if your <code class="language-plaintext highlighter-rouge">gradle-wrapper.properties</code> uses a HTTPS URL.</p>
<p>Gradle distributions are served from the following URLs:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">services.gradle.org</code></li>
<li><code class="language-plaintext highlighter-rouge">downloads.gradle.org</code></li>
<li><code class="language-plaintext highlighter-rouge">downloads.gradle-dn.com</code></li>
</ul>
<p>If your organization mirrors Gradle distributions from any of these URLs, you should check that your mirror is using HTTPS.</p>
<h3 id="other-gradle-software">Other Gradle software</h3>
<p>Other Gradle, Inc. produced software is published to an Artifactory repository, such as the Gradle Tooling API. Most builds do not use this repository unless they are building tooling that integrates with Gradle (like IntelliJ IDEA).</p>
<p>The Gradle Artifactory repository is available at <code class="language-plaintext highlighter-rouge">repo.gradle.org</code>.</p>
<h2 id="gradle-build-tool">Gradle Build Tool</h2>
<p><a href="https://docs.gradle.org/6.0-rc-1/release-notes.html#security">Gradle 6.0</a> deprecates the use of HTTP in build scripts to download resources and artifacts without an an explict opt-in.</p>
<p>For users that require the use of HTTP, Gradle has several new APIs to continue to allow HTTP on a case-by-case basis.</p>
<h2 id="timeline">Timeline</h2>
<p>To ease the transition for our users, this change is coming in a few phases.</p>
<table>
<thead>
<tr>
<th>When</th>
<th>What’s changing?</th>
</tr>
</thead>
<tbody>
<tr>
<td><del>October 29th, 2019</del></td>
<td>Gradle will begin redirecting from HTTP to HTTPS.</td>
</tr>
<tr>
<td><del>November 14th, 2019</del></td>
<td>Disable HTTP for 24 hours and permanently drop support for TLSv1. <a href="https://status.gradle.com/incidents/q2vq49fndxg0">Postmortem</a>.</td>
</tr>
<tr>
<td><del>January 15th, 2020</del></td>
<td>HTTP requests to Gradle resources will be <em>denied</em>. Only HTTPS will be supported.</td>
</tr>
<tr>
<td><del>February 15th, 2020</del></td>
<td>Enable <a href="https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security">HSTS</a> for <code class="language-plaintext highlighter-rouge">gradle.com</code> and <code class="language-plaintext highlighter-rouge">gradle.org</code>. This change will only impact browsers.</td>
</tr>
</tbody>
</table>
<h2 id="users-most-likely-to-be-impacted">Users most likely to be impacted</h2>
<p>As a part of this effort, we spent some time analyzing the data from our CDN logs to determine the size of the impact
this change would have on our users.</p>
<p>These percentages represent a sampling of our traffic over a 72 hour period.</p>
<table>
<thead>
<tr>
<th>Service</th>
<th style="text-align: right">HTTP %</th>
</tr>
</thead>
<tbody>
<tr>
<td>plugins.gradle.org/m2/*</td>
<td style="text-align: right">0.72%</td>
</tr>
<tr>
<td>services.gradle.org</td>
<td style="text-align: right">5.77%</td>
</tr>
<tr>
<td>downloads.gradle.org</td>
<td style="text-align: right">23.87%</td>
</tr>
<tr>
<td>downloads.gradle-dn.com</td>
<td style="text-align: right">9.76%</td>
</tr>
<tr>
<td>repo.gradle.org</td>
<td style="text-align: right">9.53%</td>
</tr>
</tbody>
</table>
<p>Breaking down the traffic for services.gradle.org by user agent, we can clearly see that
users of JFrog’s Artifactory are most likely to be impacted by this change.</p>
<p><img src="/images/http-decommission/HTTP_by_agent_services_gradle_org.png" alt="HTTP by Agent for services.gradle.org" /></p>
<p>Similarly, we can see while our biggest user of <code class="language-plaintext highlighter-rouge">services.gradle.org</code> is a Java user agent, Artifactory is our
second largest and is more likely to be using <code class="language-plaintext highlighter-rouge">HTTP</code> than any other User Agent.</p>
<p><img src="/images/http-decommission/HTTP_vs_HTTPS_for_services_gradle_org.png" alt="HTTP vs HTTPS for services.gradle.org by Agent" /></p>
<p>Using this data, we’ve determined that 16% of all Nexus requests, and 11% of all Artifactory requests are using HTTP instead of HTTPS.</p>
<p><strong>We recommend that you audit your corporate artifact mirrors to ensure that they are using HTTPS instead of HTTP.</strong></p>
<h2 id="why-are-we-doing-this">Why are we doing this?</h2>
<p><a href="https://medium.com/@jonathan.leitschuh/want-to-take-over-the-java-ecosystem-all-you-need-is-a-mitm-1fc329d898fb?source=friends_link&sk=3c99970c55a899ad9ef41f126efcde0e"><img src="/images/http-decommission/build_mitm.png" alt="mitm_build" /></a></p>
<p>At the beginning of June 2019, before joining the Gradle team, I
<a href="https://medium.com/@jonathan.leitschuh/want-to-take-over-the-java-ecosystem-all-you-need-is-a-mitm-1fc329d898fb?source=friends_link&sk=3c99970c55a899ad9ef41f126efcde0e">publicly disclosed my research</a>
into how many of the most popular projects across the JVM ecosystem had been resolving their dependencies over HTTP
instead of HTTPS.</p>
<p>When <a href="https://www.sonatype.com/">Sonatype</a>, the maintainers of <a href="https://search.maven.org/">Maven Central</a>,
analyzed their traffic over a month, they determined that 25% of Maven Central downloads were still using HTTP.</p>
<p>As a result of these findings, Gradle is participating in an industry-wide initiative to decommission support for HTTP from all major artifact servers starting on or near January 15th, 2020.</p>
<p>As of the publication of this blog post, these organizations are also participating and have posted announcements:</p>
<ul>
<li><a href="https://central.sonatype.org/articles/2019/Apr/30/http-access-to-repo1mavenorg-and-repomavenapacheorg-is-being-deprecated/">Sonatype Maven Central</a></li>
<li><a href="https://jfrog.com/blog/secure-jcenter-with-https/">JFrog JCenter</a></li>
<li><a href="https://spring.io/blog/2019/09/16/goodbye-http-repo-spring-use-https">Pivotal Spring</a></li>
</ul>
Introducing the Swift plugins2019-09-05T00:00:00-04:00https://blog.gradle.org/introducing-the-swift-pluginsDaniel Lacasse
<p>This post introduces some new native plugins that we’ve been working on that can build Swift libraries and applications.
They work on both macOS and Linux with the <a href="https://swift.org/download/">official Swift compiler</a>.</p>
<p>The plugins take advantage of the many features baked into Gradle core, such as a rich dependency management engine as well as <a href="https://blog.gradle.org/introducing-source-dependencies">source dependencies</a>, <a href="https://blog.gradle.org/introducing-gradle-build-cache">build cache</a>, <a href="https://blog.gradle.org/introducing-composite-builds">composite builds</a>, <a href="https://guides.gradle.org/using-the-worker-api/">finer grained parallel execution</a>, <a href="https://gradle.com/build-scans">build scans</a>, and more.
Note that, unfortunately, the plugins can’t be used out of the box to build iOS application and libraries.</p>
<p>If you’d like to provide feedback, please <a href="#feedback">get in touch with us</a>.</p>
<h2 id="building-an-application">Building an application</h2>
<p>You can find all of the samples from this post in the <a href="https://github.com/gradle/native-samples">Gradle native samples</a> GitHub repository.
Let’s look at building a <a href="https://github.com/gradle/native-samples/tree/master/swift/application">simple application</a>.</p>
<!--break-->
<p>The build script should look familiar to anyone who has used Gradle’s Java plugins:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s1">'swift-application'</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This application has no dependencies, and the Swift source files are located in the default location: the <code class="language-plaintext highlighter-rouge">src/main/swift</code> directory.
Since this is Gradle, you can easily configure the source locations to match whatever layout your project requires, including the <a href="https://github.com/gradle/native-samples/tree/master/swift/swift-package-manager">common Swift package manager pattern</a>.</p>
<p>Here’s the result of running <code class="language-plaintext highlighter-rouge">./gradlew assemble</code> on this sample:</p>
<!-- https://asciinema.org/a/YqBffKJXufvMHgajXFZ0KtTkg -->
<p><img src="/images/introducing-the-swift-plugins/01-application.gif" alt="./gradlew assemble" /></p>
<p>Take a look at the build scan for this build to see what happened in more detail.</p>
<p><a href="https://scans.gradle.com/s/3l6xcewtc365k/timeline"><img src="/images/introducing-the-swift-plugins/01-compile-debug-swift-timeline-scan.png" alt="compile debug timeline" /></a></p>
<p>The plugins automatically find the compiler, linker and other tools to build the application.
The result ends up installed in the <code class="language-plaintext highlighter-rouge">build/install</code> directory ready to run.</p>
<h2 id="ide-support">IDE support</h2>
<p>Xcode is currently supported for Swift projects.
You can just run <code class="language-plaintext highlighter-rouge">./gradlew openXcode</code>.
Gradle will generate the workspace and open the generated workspace in Xcode.
Support for other IDEs will be gradually added.</p>
<p>Here is the result of running <code class="language-plaintext highlighter-rouge">./gradlew openXcode</code> on the sample:</p>
<p><img src="/images/introducing-the-swift-plugins/02-xcode.gif" alt="./gradlew xcode" /></p>
<p>This is how the workspace looks in Xcode:</p>
<p><img src="/images/introducing-the-swift-plugins/02-xcode-ide.png" alt="Xcode integration" /></p>
<h2 id="dependencies">Dependencies</h2>
<p>The plugin uses Gradle’s <a href="https://docs.gradle.org/current/userguide/artifact_dependencies_tutorial.html">dependency management</a> features, just like other plugins such as the Java or Android plugins.
This means, for example, transitive dependencies work just fine.</p>
<p>Let’s add a dependency on a library to the application. In <a href="https://github.com/gradle/native-samples/tree/master/swift/transitive-dependencies">this sample</a>, the library and it’s dependencies are built before compiling and linking the application.
You don’t have to add the transitive library anywhere manually to your build script.</p>
<p>Here is the result of running <code class="language-plaintext highlighter-rouge">./gradlew build</code> on the sample:</p>
<!-- https://asciinema.org/a/wSgE9HkNYBKuIkaYo0O8InESA -->
<p><img src="/images/introducing-the-swift-plugins/03-dependency-management.gif" alt="./gradle build" /></p>
<p>Take a look at the build scan for this build to see what happened in more detail.</p>
<p><a href="https://scans.gradle.com/s/sb7fhisy6jrcq/timeline"><img src="/images/introducing-the-swift-plugins/03-transitive-swift-dependencies-timeline-scan.png" alt="transitive dependencies" /></a></p>
<p>Here is how this project looks in Xcode:</p>
<p><img src="/images/introducing-the-swift-plugins/03-xcode-ide.png" alt="Xcode integration" /></p>
<h2 id="tests">Tests</h2>
<p>XCTest testing is supported out of the box.
Here is a <a href="https://github.com/gradle/native-samples/tree/master/swift/simple-library">sample that uses XCTest</a>.
It will locate the XCTest Framework inside your Swift tool chain installation.</p>
<p>The build script applies the <a href="https://docs.gradle.org/current/userguide/xctest_plugin.html">XCTest plugin</a> which configures your build to generate an XCTest bundle on macOS or an XCTest executable on Linux:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s1">'xctest'</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here is the result of running <code class="language-plaintext highlighter-rouge">./gradlew check</code>. Gradle locates the XCTest module inside the tool chain installation, compiles the Swift source and tests, and then runs the tests:</p>
<!-- https://asciinema.org/a/oVBqHXUvNFRvI1EhO36pACUTn -->
<p><img src="/images/introducing-the-swift-plugins/04-xctest.gif" alt="./gradlew check" /></p>
<p><a href="https://scans.gradle.com/s/6j6luunc2h4tw"><img src="/images/introducing-the-swift-plugins/04-xctest-timeline-scan.png" alt="build scan for math check" /></a></p>
<p>Gradle has a deep understanding of this testing framework.
It can <a href="https://docs.gradle.org/current/userguide/swift_testing.html#sec:swift_test_reporting">generate JUnit and HTML test results report</a> as well as supports the <code class="language-plaintext highlighter-rouge">LinuxMain.swift</code> convention for declaring the XCTest Linux executable entry point.
It also <a href="https://docs.gradle.org/current/userguide/swift_testing.html#sec:swift_test_filtering">supports basic test filtering</a>.
Build scan support, parallel execution and improved filtering for XCTest will be added later.</p>
<h2 id="fast-builds">Fast builds</h2>
<p>The plugins can produce debug and release builds of the application or library using Gradle’s new variant-aware dependency management, so that debug builds are compiled and linked against debug library binaries, and release builds are compiled and linked against release library binaries.
When you build the debug build, which is the default, Gradle builds <em>only</em> the debug builds of the libraries that you need, rather than building everything.</p>
<p>Developer and CI builds are fast.
Swift compilation is a cacheable task, so you can avoid unnecessary and long compilation times when using the <a href="https://docs.gradle.org/current/userguide/build_cache.html">build cache</a>.
<a href="https://gradle.com/build-cache">Gradle Enterprise</a> comes with a build cache backend.
You don’t need to use the <code class="language-plaintext highlighter-rouge">--parallel</code> option as Gradle does incremental and parallel compilation and linking by default.</p>
<p>Let’s run some clean builds that use the build cache:</p>
<!-- https://asciinema.org/a/OitpkgOf3Rz8EZr8YlipcExQ1 -->
<p><img src="/images/introducing-the-swift-plugins/05-build-cache.gif" alt="./gradlew assemble with build cache" /></p>
<p>You can see that the second build is faster, as the result is fetched from the build cache rather than recompiled.
Build scans for <a href="https://gradle.com/s/3p3jigoiwmpxk">non-cached build</a> and <a href="https://gradle.com/s/h5xm2wcujfv2m">cached build</a>.</p>
<p><img src="/images/introducing-the-swift-plugins/05-cached-vs-non-cached.png" alt="cached vs non-cached assemble" /></p>
<h2 id="composite-builds">Composite builds</h2>
<p>Composite builds also work the same as in Java or C++ projects.
This <a href="https://github.com/gradle/native-samples/tree/master/swift/composite-build">sample using a composite build</a> combines builds so they can be worked on together:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">rootProject</span><span class="o">.</span><span class="na">name</span> <span class="o">=</span> <span class="s1">'app'</span>
<span class="n">includeBuild</span> <span class="s1">'utilities-library'</span>
<span class="n">includeBuild</span> <span class="s1">'list-library'</span>
</code></pre></div></div>
<p>Here’s the result in Xcode.
The application and the libraries it uses are available to edit, build and test together:</p>
<p><img src="/images/introducing-the-swift-plugins/07-xcode-ide.png" alt="Xcode integration" /></p>
<h3 id="operating-system-and-source-compatibility">Operating System and Source Compatibility</h3>
<p>Native projects can also configure the operating system (a.k.a target machines) for Swift components.</p>
<p>Each component script block (e.g. <a href="https://docs.gradle.org/current/dsl/org.gradle.language.swift.SwiftApplication.html">application</a>, <a href="https://docs.gradle.org/current/dsl/org.gradle.language.swift.SwiftLibrary.html">library</a>, <a href="https://docs.gradle.org/current/dsl/org.gradle.nativeplatform.test.xctest.SwiftXCTestSuite.html">xctest</a>) can configure the <code class="language-plaintext highlighter-rouge">targetMachines</code> property.
The configured target machines will participate in the <a href="https://docs.gradle.org/current/userguide/building_swift_projects.html#sec:introducing_build_variants">build variant</a> for the machine compatible with the build host.
For example, the following specifies that the application is only buildable on Linux machines.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s2">"swift-application"</span>
<span class="o">}</span>
<span class="n">application</span> <span class="o">{</span>
<span class="n">targetMachines</span> <span class="o">=</span> <span class="o">[</span><span class="n">machines</span><span class="o">.</span><span class="na">linux</span><span class="o">.</span><span class="na">x86_64</span><span class="o">]</span>
<span class="o">}</span>
</code></pre></div></div>
<p>On a Linux machine, a variant of all the tasks will be created.
As shown on the screencast below, <a href="https://gradle.com/s/h6nwxuvy7hdso">the <code class="language-plaintext highlighter-rouge">assemble</code> task compiles the <code class="language-plaintext highlighter-rouge">debug</code> variant</a>, also known as the development binary.</p>
<!-- https://asciinema.org/a/PpSKXcBRFbY9YNKr2PCeQdbmt -->
<p><img src="/images/introducing-the-swift-plugins/09-building-swift-application-on-macos.gif" alt="Building Swift Application on macOS" /></p>
<p>The other variant <code class="language-plaintext highlighter-rouge">release</code> is buildable via the <code class="language-plaintext highlighter-rouge">linkReleaseX86-64</code> tasks.</p>
<p>On other operating systems, Gradle won’t build anything since no target machine is compatible with the build host.
It’s worth noting that none of the tasks mentioned previously will be created as the variants aren’t buildable.
Instead, invoking the assemble task will simply print an informative message as shown in the screencast below.</p>
<p><img src="/images/introducing-the-swift-plugins/09-building-swift-application-on-linux.png" alt="Building Swift Application on Linux" /></p>
<h3 id="source-dependencies">Source dependencies</h3>
<p>Support for source dependencies was recently announced in the <a href="https://blog.gradle.org/introducing-source-dependencies">Introducing source dependencies</a> blog post.</p>
<p>In short, source dependencies allow Gradle to use other projects as dependencies directly from source.
It is common practice for Swift developers to build dependencies from source.
Gradle takes care of all the wiring required to build sources before building the consumer.</p>
<p>If the <a href="https://github.com/gradle/native-samples/tree/master/swift/source-dependencies">source dependency build is a Gradle build</a>, the experience will be similar to a composite <a href="https://docs.gradle.org/current/userguide/composite_builds.html">included build</a>.
On the other hand, if the source dependency is built by another build system, Gradle can inject configuration code to execute the right build commands and map the artifacts into a Gradle build.</p>
<h2 id="documentation">Documentation</h2>
<p>If you are familiar with the <a href="https://github.com/gradle/native-samples">native samples repository</a>, chances are you are already familiar with the samples demonstrating what can already be achieved with Swift plugins.
With Gradle 5.6, we released additional documentation to help new users get on-board with Swift development.
We suggest reading the <a href="https://docs.gradle.org/current/userguide/building_swift_projects.html">Building Swift Projects</a> and <a href="https://docs.gradle.org/current/userguide/swift_testing.html">Testing Swift Projects</a> chapters of the user guide to get started.</p>
<p>We have also introduced <a href="https://docs.gradle.org/current/userguide/plugin_reference.html#native_languages">reference chapters for each Swift plugin</a>.
You can also read the <a href="https://docs.gradle.org/current/userguide/xcode_plugin.html">Xcode plugin reference chapter</a> to learn how to use the Xcode IDE during your Swift development.</p>
<p><a name="feedback"></a></p>
<h2 id="providing-feedback">Providing Feedback</h2>
<p>These plugins are a work in progress and have some limitations. For example, model elements such as debuggability don’t allow any configuration yet and you’ll need to resort to using compiler/linker flags.</p>
<p>We’ll continue to improve these plugins, make them stable, and eventually support iOS application.</p>
<p>Please try these plugins out and let us know what you think.
The easiest way to get started is to clone the <a href="https://github.com/gradle/native-samples">native samples repository</a> and follow the instructions.
Our samples use a <a href="https://gradle.org/nightly/">Gradle nightly build</a>, so you’ll see the latest and greatest developments there.</p>
<p>We’d love to hear what you think works well, what’s confusing, and what is missing that would block you from using Gradle to build Swift software.
You can also leave feedback on the <a href="https://discuss.gradle.org/">Gradle forums</a> or raise issues on the <a href="https://github.com/gradle/gradle-native">Gradle native</a> GitHub repository.</p>
Update on the new C++ plugins2019-06-04T00:00:00-04:00https://blog.gradle.org/update-on-the-new-cpp-pluginsDaniel Lacasse
<p>As the new C++ plugins are getting more attention from early adopters, we want to update everyone on their progress since our previous <a href="https://blog.gradle.org/introducing-the-new-cpp-plugins">Introducing the new C++ plugins</a> post.</p>
<p>The last few months have seen several new features, new and expanded documentation, and improvements in IDE support.</p>
<p>If you’d like to provide feedback, please <a href="#feedback">get in touch with us</a>.</p>
<h2 id="new-features">New Features</h2>
<p>Since the last blog post, we introduced three major new features: starter templates for C++ projects, target machine modeling and source dependencies.</p>
<h3 id="starter-templates-for-c-projects">Starter templates for C++ projects</h3>
<p>Starting a new C++ project is now easier than ever.</p>
<p>Thanks to C++ starter templates for the <a href="https://docs.gradle.org/current/userguide/build_init_plugin.html"><code class="language-plaintext highlighter-rouge">init</code> task added by the Build Init Plugin</a>, you can initialize a <a href="https://docs.gradle.org/current/userguide/build_init_plugin.html#sec:cppapplication_">C++ Application</a> or <a href="https://docs.gradle.org/current/userguide/build_init_plugin.html#sec:cpplibrary_">C++ Library</a>.
Here’s an example of initializing a C++ application using the Kotlin DSL with the Build Init Plugin:</p>
<p><img src="/images/update-on-the-new-cpp-plugins/03-starting-cpp-application-with-build-init.gif" alt="Starting C++ Application with the `init` task" /></p>
<h3 id="operating-system-and-architecture-support">Operating System and Architecture Support</h3>
<p>Native projects can now also configure the operating system and architecture (a.k.a target machines) on C++ components.</p>
<p>Each component script block (e.g. <a href="https://docs.gradle.org/current/dsl/org.gradle.language.cpp.CppApplication.html">application</a>, <a href="https://docs.gradle.org/current/dsl/org.gradle.language.cpp.CppLibrary.html">library</a>, <a href="https://docs.gradle.org/current/dsl/org.gradle.nativeplatform.test.cpp.CppTestSuite.html">unitTest</a>) can configure the <code class="language-plaintext highlighter-rouge">targetMachines</code> property.
The configured target machines will participate in the <a href="https://docs.gradle.org/current/userguide/building_cpp_projects.html#sec:introducing_build_variants">build variant</a> for the machine compatible with the build host.
For example, the following configuration of <code class="language-plaintext highlighter-rouge">targetMachines</code> on a Windows machines results in dedicated task for both x86 and x86-64 architectures.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s2">"cpp-application"</span>
<span class="o">}</span>
<span class="n">application</span> <span class="o">{</span>
<span class="n">targetMachines</span> <span class="o">=</span> <span class="o">[</span><span class="n">machines</span><span class="o">.</span><span class="na">windows</span><span class="o">.</span><span class="na">x86</span><span class="o">,</span> <span class="n">machines</span><span class="o">.</span><span class="na">windows</span><span class="o">.</span><span class="na">x86_64</span><span class="o">]</span>
<span class="o">}</span>
</code></pre></div></div>
<p>On a Windows machine, a variant of all the tasks will be created for both x86 and x86-64 architectures.
As shown on the screencast below, the <code class="language-plaintext highlighter-rouge">assemble</code> task compiles the debug x86-64 variant, also known as the development binary.</p>
<p><img src="/images/update-on-the-new-cpp-plugins/04-building-cpp-application-on-windows.gif" alt="Building C++ Application on Windows" /></p>
<p>The other variants <code class="language-plaintext highlighter-rouge">{debug, x86}</code>, <code class="language-plaintext highlighter-rouge">{release, x86}</code>, and <code class="language-plaintext highlighter-rouge">{release, x86-64}</code> are buildable via the <code class="language-plaintext highlighter-rouge">linkDebugX86</code>, <code class="language-plaintext highlighter-rouge">linkReleaseX86</code>, and <code class="language-plaintext highlighter-rouge">linkReleaseX86-64</code> tasks respectively.</p>
<p>On other operating systems, Gradle won’t build anything since no target machine is compatible with the build host.
It’s worth noting that none of the tasks mentioned previously will be created as the variants aren’t buildable.
Instead, invoking the assemble task will simply print an informative message as shown in the screencast below.</p>
<p><img src="/images/update-on-the-new-cpp-plugins/05-building-cpp-application-on-macos.gif" alt="Building C++ Application on macOS" /></p>
<h3 id="source-dependencies">Source dependencies</h3>
<p>Support for source dependencies was an unannounced feature until only recently. You can read all about it in the <a href="https://blog.gradle.org/introducing-source-dependencies">Introducing source dependencies</a> post.</p>
<p>In short, source dependencies allow Gradle to use other projects as dependencies directly from source.
It is common practice for C++ developers to build dependencies from source. Gradle takes care of all the wiring required to build sources before building the consumer.</p>
<p>If the <a href="https://github.com/gradle/native-samples/tree/master/cpp/source-dependencies">source dependency build is a Gradle build</a>, the experience will be similar to a composite <a href="https://docs.gradle.org/current/userguide/composite_builds.html">included build</a>.
On the other hand, if the <a href="https://github.com/gradle/native-samples#application-and-libraries-built-by-cmake-cmake-source-dependencies">source dependency is built by another build system, say CMake</a>, Gradle can inject configuration code to execute the right build commands and map the artifacts into a Gradle build.</p>
<h2 id="documentation">Documentation</h2>
<p>On the path leading to Gradle 5.5, we released quite a few more samples demonstrating what can already be achieved with the C++ plugins.</p>
<p>Some of those samples present interesting use cases such as <a href="https://github.com/gradle/native-samples/tree/master/cpp/cmake-library">integrating with CMake</a> and <a href="https://github.com/gradle/native-samples/tree/master/cpp/provisionable-tool-chains">how to automatically download and use tool chains from the vendor’s website such as Clang</a>.</p>
<p><img src="/images/update-on-the-new-cpp-plugins/02-integration-with-cmake.gif" alt="Integration with CMake" /></p>
<p>In Gradle 5.5, we released all new C++ documentation.
We suggest reading the <a href="https://docs.gradle.org/current/userguide/building_cpp_projects.html">Building C++ Projects</a> and <a href="https://docs.gradle.org/current/userguide/cpp_testing.html">Testing C++ Projects</a> chapters to get started.</p>
<p>We are also happy to introduce <a href="https://docs.gradle.org/current/userguide/plugin_reference.html#native_languages">reference chapters for each native plugin</a> and also for <a href="https://docs.gradle.org/current/userguide/plugin_reference.html#ide_integration">Visual Studio and Xcode plugins</a>.</p>
<h2 id="ide-support">IDE support</h2>
<p>The <a href="https://docs.gradle.org/current/userguide/visual_studio_plugin.html">Visual Studio Plugin</a> supports generating solution files for projects using the new C++ plugin.</p>
<p><img src="/images/update-on-the-new-cpp-plugins/06-visual-studio-ide.png" alt="Visual Studio IDE" /></p>
<p>JetBrains built on top of their famous native Gradle imported from IntelliJ and release <a href="https://www.jetbrains.com/help/clion/gradle-support.html">support for opening Gradle native project</a> directly inside Clion using the tooling API.</p>
<p>The experience is just as a user would expect. You can build, debug and enjoy code completion while writing code:</p>
<p><img src="/images/update-on-the-new-cpp-plugins/07-clion-ide.gif" alt="Clion IDE" /></p>
<p>The new C++ plugins also supports IDE development with Xcode.</p>
<p><a name="feedback"></a></p>
<h2 id="providing-feedback">Providing feedback</h2>
<p>These plugins are a work in progress and have some limitations.</p>
<p>For example, model elements such as debuggability don’t allow any configuration yet.
You need to resort to using compiler/linker flags.</p>
<p>We’ll continue to improve these plugins, make them stable, and eventually deprecate the software model plugins.</p>
<p>Please try these plugins out and let us know what you think.
The easiest way to get started is to clone the <a href="https://github.com/gradle/native-samples">native samples repository</a> and follow the instructions.
Our samples use a <a href="https://gradle.org/nightly/">Gradle nightly build</a>, so you’ll see the latest and greatest developments there.</p>
<p>We’d love to hear what you think works well, what’s confusing, and what is missing that would block you from using Gradle to build C++ software.
You can also leave feedback on the <a href="https://discuss.gradle.org/">Gradle forums</a> or raise issues on the <a href="https://github.com/gradle/gradle-native">Gradle native</a> GitHub repository.</p>
Automatic task execution in Buildship2019-05-02T00:00:00-04:00https://blog.gradle.org/buildship-sync-task-execDonat Csikos
<p>If you work with Eclipse you are probably familiar with Buildship, the Eclipse plugins for Gradle.
The Buildship 3.1 release allows you to <a href="https://github.com/eclipse/buildship/issues/265">run tasks upon project synchronization</a> and <a href="https://github.com/eclipse/buildship/issues/266">auto build</a>, the two most highly voted issues on Github.
In this post, we’re going to summarize why this is an essential feature for many and how can you make use of it.</p>
<h3 id="extending-project-synchronization">Extending project synchronization</h3>
<p>The Buildship project synchronization functionality imports the Gradle projects into the workspace and configures them to work with the Java toolchain.
That - of course - is just the tip of the iceberg. There are many other tools and frameworks out there, and Buildship can’t provide configuration for all.
Since 3.0, there’s a <a href="(https://projects.eclipse.org/projects/tools.buildship/releases/3.0.0#project-configurators)">public API</a> to add new integrations, but that requires writing Eclipse plugins.
Most users only want to run custom tasks to generate or update configuration files.
Running the tasks manually after each change can be frustrating and error-prone.
Automatically running the tasks upon project synchronization helps the developers to stay in the flow.</p>
<h3 id="how-it-works">How it works</h3>
<p>To use the new feature, you’ll need Buildship 3.1 and a project using Gradle 5.4 and above.
In Gradle 5.4 we introduced a new attribute in the <code class="language-plaintext highlighter-rouge">eclipse</code> plugin:</p>
<figure class="highlight"><pre><code class="language-groovy" data-lang="groovy"><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s1">'eclipse'</span>
<span class="o">}</span>
<span class="n">task</span> <span class="n">generateCustomConfig</span> <span class="o">{</span>
<span class="n">doLast</span> <span class="o">{</span>
<span class="n">println</span> <span class="s2">"Generating custom configuration..."</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">eclipse</span> <span class="o">{</span>
<span class="n">synchronizationTasks</span> <span class="n">generateCustomConfig</span>
<span class="o">}</span></code></pre></figure>
<p>That’s it.
When you import or synchronize the project, you’ll see the tasks being executed.</p>
<p><img src="/images/buildship-sync-task-exec.png" alt="Executing tasks upon synchronization" /></p>
<p>Note, that the synchronization task is declared with a task reference.
You can actually use different types here: strings specifying the task paths, a list of tasks, and more.
Essentially, you can use any <a href="https://docs.gradle.org/current/javadoc/org/gradle/api/Task.html#dependencies">task dependency types</a>.</p>
<h3 id="executing-tasks-during-eclipse-build">Executing tasks during Eclipse build</h3>
<p>Along with the feature above, we also added the option to run tasks every time the user changes a resource in the workspace, and the Eclipse build is triggered.
This feature is useful to execute small code generator and validator tasks.
The syntax is very similar:</p>
<figure class="highlight"><pre><code class="language-groovy" data-lang="groovy"><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s1">'eclipse'</span>
<span class="o">}</span>
<span class="n">task</span> <span class="n">generateCode</span> <span class="o">{</span>
<span class="n">doLast</span> <span class="o">{</span>
<span class="n">println</span> <span class="s1">'Generating some code...'</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">eclipse</span> <span class="o">{</span>
<span class="n">autoBuildTasks</span> <span class="n">generateCode</span>
<span class="o">}</span></code></pre></figure>
<h3 id="conclusion">Conclusion</h3>
<p>This new feature will enable a lot of developers and build authors to provide a smoother work experience within the IDE.
Let us know what you think about it and how would you make use if in your own setup.</p>
Introducing Gradle Module Metadata2019-03-07T00:00:00-05:00https://blog.gradle.org/gradle-metadata-1.0Cédric Champeau
<p>Gradle Module Metadata reaches 1.0 in Gradle 5.3 and here we explain why you should be as excited as we are!</p>
<p>Gradle Module Metadata was created to solve many of the problems that have plagued dependency management for years, in particular, but not exclusively, in the Java ecosystem.
It is especially important because POM files (or Ivy files) are simply not rich enough to describe the reality of software nowadays where you might need to distinguish between binaries for different platforms or choose one particular implementation of an API when more than one is available.</p>
<p>We will describe more examples later in this post.
Some issues may have workarounds, but where those workarounds are hacky ones or even error prone.
For example, did you realize that these are problematic: using classifiers for different Java versions, exclusions to avoid a particular logger binding, or adding first level dependencies just because you need to override a particular version?</p>
<p>Gradle Module Metadata 1.0 is an answer to those problems and the first step towards better dependency management throughout our industry.</p>
<h3 id="what-does-it-allow-in-practice">What does it allow in practice?</h3>
<p>Have you ever cursed when you had both <code class="language-plaintext highlighter-rouge">guava-jdk5</code> and <code class="language-plaintext highlighter-rouge">guava-jdk8</code> on the classpath, and your application only worked because of lucky ordering of entries?
Have you ever faced the problem of having different SLF4J bindings and only noticed at runtime?
That’s because these libraries have different <em>variants</em> that can’t be described properly by existing metadata formats.
How is a build tool even supposed to understand the difference between a <code class="language-plaintext highlighter-rouge">jdk8</code> JAR, a <code class="language-plaintext highlighter-rouge">sources</code> one, or even an <code class="language-plaintext highlighter-rouge">all</code> one?
Gradle Module Metadata is designed to explain the difference in such a way that consumers can express more precise requirements.
For example, a consumer can specifically ask for something they can use with JDK 8.
And in the case of SLF4J, the build tool will recognize that the Log4J binding is mutually exclusive with the <code class="language-plaintext highlighter-rouge">java.util.logging</code> one.</p>
<p>The whole idea is to support variant-aware dependency management, which is based on a description of the variants of a component — such as the main binaries, source packages, platform-specific binaries, and so on — and their corresponding dependencies.</p>
<p>Some of our partners have been using Gradle metadata for months now. <a href="https://github.com/JetBrains/kotlin-native/">Kotlin native</a>, for example, is using Gradle Module Metadata to represent the different binaries you can get when compiling a Kotlin project to different architectures.
Google is using <a href="https://developer.android.com/studio/build/build-variants">variant-aware dependency management</a>, but lacked an “external model” for it. Gradle Module Metadata is that external model and will allow the proper publication of Android Archives (AARs).</p>
<p>Those are just examples, but these issues and many more can be solved by leveraging Gradle metadata. As more library authors adopt Gradle Module Metadata, our industry will solve more problems as a whole.</p>
<h4 id="how-gradle-module-metadata-affects-you">How Gradle Module Metadata affects you</h4>
<p>Gradle Module Metadata 1.0 enables fine-grained dependency resolution for all Gradle users.
Starting with Gradle 5.3, if you are a consumer and the library you use has Gradle metadata published, Gradle will automatically consume any Gradle metadata that is published to Maven or Ivy repositories.</p>
<p>However, Gradle 5.3 will not automatically <em>publish</em> it by default — that will come in 6.0. You can publish Gradle Module Metadata today, but you have to opt into its publication by using either the Maven Publish or Ivy Publish plugins and enabling the experimental publishing feature by adding the following line to your settings script:</p>
<h4 id="settingsgradlekts">settings.gradle(.kts)</h4>
<figure class="highlight"><pre><code class="language-groovy" data-lang="groovy"><span class="n">enableFeaturePreview</span><span class="o">(</span><span class="s2">"GRADLE_METADATA"</span><span class="o">)</span></code></pre></figure>
<h4 id="how-does-gradle-module-metadata-affect-maven-or-antivy-builds">How does Gradle Module Metadata affect Maven or Ant+Ivy builds?</h4>
<p>Nothing changes for Maven and Ivy consumers: if you have opted into publishing Gradle Module Metadata, the corresponding file is published alongside the POM file (if you’re publishing to a Maven repository) or Ivy file (if you’re publishing to an Ivy repository).
Be aware that the mapping from a build component to Maven and Ivy metadata is <em>lossy</em>: for example you don’t know what Java version was used to build something, so it makes it impossible for consumers to know if they are compatible or not beforehand. Another example is when you use Gradle specific features like <a href="https://docs.gradle.org/current/userguide/declaring_dependencies.html#sub:declaring_dependency_rich_version">rich versions</a>. We do our best to map them to concepts in Maven or Ivy, but information will still be lost in the process due to the limitations of their metadata formats.</p>
<p>Note that Gradle 5.2 introduced warnings emitted by the publishing plugins when it knows the mapping to be lossy or problematic for other build tools.</p>
<h3 id="links-and-references">Links and references</h3>
<p>Gradle Module Metadata is a JSON file whose extension is <code class="language-plaintext highlighter-rouge">.module</code>.
Each file describes a single software component with zero or more variants. Here’s the content of an example metadata file for a “com.acme:client:1.0-SNAPSHOT” component with several variants:</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="nl">"formatVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0"</span><span class="p">,</span><span class="w">
</span><span class="nl">"component"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"com.acme"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"client"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0-SNAPSHOT"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"org.gradle.status"</span><span class="p">:</span><span class="w"> </span><span class="s2">"integration"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"createdBy"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"gradle"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5.3"</span><span class="p">,</span><span class="w">
</span><span class="nl">"buildId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4wqjtkcv2fbmjjsewyu66wbvfq"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"variants"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"apiElements"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"org.gradle.dependency.bundling"</span><span class="p">:</span><span class="w"> </span><span class="s2">"external"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.gradle.jvm.version"</span><span class="p">:</span><span class="w"> </span><span class="mi">11</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.gradle.usage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"java-api-jars"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"com.mycompany"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"core"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"requires"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2.5"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"files"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"client-1.0-SNAPSHOT.jar"</span><span class="p">,</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"client-1.0-SNAPSHOT.jar"</span><span class="p">,</span><span class="w">
</span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">539</span><span class="p">,</span><span class="w">
</span><span class="nl">"sha1"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1f94fe53d33babdc9de537bb3a0108dbc0e25e4b"</span><span class="p">,</span><span class="w">
</span><span class="nl">"md5"</span><span class="p">:</span><span class="w"> </span><span class="s2">"6364cdd9923e1eda9b328bc80f93969c"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"runtimeElements"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"org.gradle.dependency.bundling"</span><span class="p">:</span><span class="w"> </span><span class="s2">"external"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.gradle.jvm.version"</span><span class="p">:</span><span class="w"> </span><span class="mi">11</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.gradle.usage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"java-runtime-jars"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.apache.commons"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"commons-lang3"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"requires"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3.8"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"com.mycompany"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="s2">"core"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"requires"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2.5"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"files"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"client-1.0-SNAPSHOT.jar"</span><span class="p">,</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"client-1.0-SNAPSHOT.jar"</span><span class="p">,</span><span class="w">
</span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">539</span><span class="p">,</span><span class="w">
</span><span class="nl">"sha1"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1f94fe53d33babdc9de537bb3a0108dbc0e25e4b"</span><span class="p">,</span><span class="w">
</span><span class="nl">"md5"</span><span class="p">:</span><span class="w"> </span><span class="s2">"6364cdd9923e1eda9b328bc80f93969c"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"shadowApiElements"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"org.gradle.dependency.bundling"</span><span class="p">:</span><span class="w"> </span><span class="s2">"shadowed"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.gradle.usage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"java-api"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"files"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"client-1.0-SNAPSHOT-all.jar"</span><span class="p">,</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"client-1.0-SNAPSHOT-all.jar"</span><span class="p">,</span><span class="w">
</span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">601730</span><span class="p">,</span><span class="w">
</span><span class="nl">"sha1"</span><span class="p">:</span><span class="w"> </span><span class="s2">"9b70e54ffdce0541701d8f855bf75e059857eb0c"</span><span class="p">,</span><span class="w">
</span><span class="nl">"md5"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3499bb6d9ccf86283854a5550135ea4a"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"shadowRuntimeElements"</span><span class="p">,</span><span class="w">
</span><span class="nl">"attributes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"org.gradle.dependency.bundling"</span><span class="p">:</span><span class="w"> </span><span class="s2">"shadowed"</span><span class="p">,</span><span class="w">
</span><span class="nl">"org.gradle.usage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"java-runtime-jars"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"files"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"client-1.0-SNAPSHOT-all.jar"</span><span class="p">,</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"client-1.0-SNAPSHOT-all.jar"</span><span class="p">,</span><span class="w">
</span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">601730</span><span class="p">,</span><span class="w">
</span><span class="nl">"sha1"</span><span class="p">:</span><span class="w"> </span><span class="s2">"9b70e54ffdce0541701d8f855bf75e059857eb0c"</span><span class="p">,</span><span class="w">
</span><span class="nl">"md5"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3499bb6d9ccf86283854a5550135ea4a"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>This file declares 4 variants, and attributes let the build tool know what they are used for.
In particular you will see here that there are 2 variants for “API” and 2 variants for “runtime”, when usually you only see one for each.
The reason is that this particular component declares an additional variant where the dependencies are <em>shadowed</em> (fat jar).
This gives the opportunity for a consumer to decide whether it wants dependencies as individual jars, or just the fatjar variant of the library.</p>
<p>If you are interested in more technical details, please refer to the <a href="https://github.com/gradle/gradle/blob/f6a98158e75a636245f70d46604fcab3152361e8/subprojects/docs/src/docs/design/gradle-module-metadata-1.0-specification.md">Gradle module metadata specification 1.0</a></p>
<p>Early adopters are welcome, feel free to <a href="https://discuss.gradle.org/t/feedback-needed-gradle-module-metadata/30842">give your feedback</a>!</p>
Developer Productivity Day 2019, Munich2019-02-05T00:00:00-05:00https://blog.gradle.org/developer-productivity-dayStefan Wolf
<p><em>This post is in German, announcing the Developer Productivity Day in Munich.</em></p>
<p>Gradle Inc. und TNG Technology Consulting GmbH laden Sie herzlich ein zum Developer Productivity Day am 19. Februar 2019 in die Arabellastraße 4a in München.</p>
<p>Wie verändert sich die Entwicklungslandschaft?
Was sind Faktoren, die heute oder in Zukunft die Software-Entwicklung stark positiv verändern werden?</p>
<p>TNG Technology Consulting zusammen mit ihren Partnern Gradle und CQSE wollen reflektieren, was heute schon möglich und morgen der Standard sein wird: in gewissen Bereichen ein Faktor 10, 100 oder sogar manchmal 1000 an Verbesserung in der Geschwindigkeit der Builds und somit der Feedback-Schleifen für Entwickler.</p>
<p>Tauschen Sie sich mit anderen aus und diskutieren mit!
Die Teilnahme ist kostenlos und Sie können sich über <a href="https://www.eventbrite.de/e/developer-productivity-day-2019-tickets-53051942789">Eventbrite</a> anmelden.
Dort finden Sie auch die Abstracts der Vorträge und weitere Informationen zu dem Event.</p>
<p><strong><a href="https://www.eventbrite.de/e/developer-productivity-day-2019-tickets-53051942789">Jetzt Anmelden!</a></strong></p>
<p>Wir freuen uns Sie persönlich beim Developer Productivity Day 2019 zu treffen!</p>
<h4 id="programm">Programm</h4>
<table>
<tbody>
<tr>
<td>13:30 - 14:00</td>
<td>Ankommen, Kennenlernen bei Kaffee und Snacks</td>
<td> </td>
</tr>
<tr>
<td>14:00 - 14:10</td>
<td>Begrüßung</td>
<td>Gerhard Müller,<br /> TNG Technology Consulting</td>
</tr>
<tr>
<td>14:10 - 14:45</td>
<td>Schnelle und zuverlässige Builds mit Gradle und Maven</td>
<td>Dr. Stefan Wolf,<br /> Gradle Inc.</td>
</tr>
<tr>
<td>14:55 - 15:30</td>
<td>Muss ich wirklich schon wieder alles testen? Test-Impact-Analyse in Forschung und Praxis</td>
<td>Dr. Elmar Jürgens,<br /> CQSE</td>
</tr>
<tr>
<td>15:30 - 16:15</td>
<td>Pause bei Kaffee und Snacks</td>
<td> </td>
</tr>
<tr>
<td>16:15 - 16:50</td>
<td>Erfahrungsbericht: Eclipse-RCP Anwendungen bauen mit Gradle</td>
<td>Andreas Turban,<br /> Vector Informatik</td>
</tr>
<tr>
<td>17:00 - 17:35</td>
<td>Reflexion täglicher Herausforderungen im Umfeld Developer Productivity</td>
<td>Andreas Schmid,<br /> TNG Technology Consulting</td>
</tr>
<tr>
<td>ab 17:35</td>
<td>Networking und Pizza</td>
<td> </td>
</tr>
</tbody>
</table>
Using build scan tags for ad-hoc root cause analysis2019-01-15T00:00:00-05:00https://blog.gradle.org/build-scans-tag-root-causeTony Robalik
<p>Recently I was helping an Android team investigate a hard-to-reproduce issue that manifested as very-long-running compilation and annotation processing tasks.
Most of their builds took only a few minutes to run, but sometimes they took up to 30 minutes!
In these cases, invariably, a build scan showed that some combination of Java compilation, Kotlin compilation, or annotation processing with Kapt, was the culprit.
The team had no real idea of how often this problem occurred for developers, and under what conditions.</p>
<p>Gradle Enterprise does not, yet, provide a way to find builds based on how long a particular task took.
However, custom <strong>tags</strong> make it easy to categorize and find builds for any condition that can be detected within a build.
By tagging all of the too long builds we could use Gradle Enterprise to understand the impact and severity, and the context in which they occur, which accelerated the debugging.</p>
<p>To achieve this, we simply added something like the following snippet to the build.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// root build.gradle</span>
<span class="n">subprojects</span> <span class="o">{</span>
<span class="n">tasks</span><span class="o">.</span><span class="na">matching</span> <span class="o">{</span> <span class="n">t</span> <span class="o">-></span>
<span class="n">t</span> <span class="k">instanceof</span> <span class="n">org</span><span class="o">.</span><span class="na">jetbrains</span><span class="o">.</span><span class="na">kotlin</span><span class="o">.</span><span class="na">gradle</span><span class="o">.</span><span class="na">tasks</span><span class="o">.</span><span class="na">KotlinCompile</span>
<span class="o">||</span> <span class="n">t</span> <span class="k">instanceof</span> <span class="n">org</span><span class="o">.</span><span class="na">jetbrains</span><span class="o">.</span><span class="na">kotlin</span><span class="o">.</span><span class="na">gradle</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">KaptWithKotlincTask</span>
<span class="o">||</span> <span class="n">t</span> <span class="k">instanceof</span> <span class="n">com</span><span class="o">.</span><span class="na">android</span><span class="o">.</span><span class="na">build</span><span class="o">.</span><span class="na">gradle</span><span class="o">.</span><span class="na">tasks</span><span class="o">.</span><span class="na">factory</span><span class="o">.</span><span class="na">AndroidJavaCompile</span>
<span class="c1">// || ... more ...</span>
<span class="o">}.</span><span class="na">configureEach</span> <span class="o">{</span>
<span class="kt">long</span> <span class="n">start</span>
<span class="n">doFirst</span> <span class="o">{</span>
<span class="n">start</span> <span class="o">=</span> <span class="n">System</span><span class="o">.</span><span class="na">currentTimeMillis</span><span class="o">()</span>
<span class="o">}</span>
<span class="n">doLast</span> <span class="o">{</span>
<span class="kt">long</span> <span class="n">duration</span> <span class="o">=</span> <span class="n">System</span><span class="o">.</span><span class="na">currentTimeMillis</span><span class="o">()</span> <span class="o">-</span> <span class="n">start</span>
<span class="k">if</span> <span class="o">(</span><span class="n">duration</span> <span class="o">></span> <span class="mi">15</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">1000</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// 15 min</span>
<span class="n">buildScan</span><span class="o">.</span><span class="na">tag</span> <span class="s2">"TooLong"</span>
<span class="n">buildScan</span><span class="o">.</span><span class="na">value</span> <span class="s2">"TooLong"</span><span class="o">,</span> <span class="n">path</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This code block will tag a build with “TooLong” if any of the tasks-of-interest took longer than 15 minutes to run.
Furthermore, it will add a custom value to the build with the task path as the value, so we will know exactly which task(s) are the culprit for any given build.
The tag or custom value can then be used as search criteria in Gradle Enterprise to find all builds that exhibited the problem.</p>
<p><img src="/images/scans_list_tag_too_long.png" alt="Scans List" /></p>
<p>This is but one example of tagging a build based on what happened.
The API for tagging builds combined with Gradle’s rich and extensive API for interrogating the build open up many possibilities.</p>
<h2 id="further-reading">Further reading</h2>
<ul>
<li>For more information on build scans, please see <a href="https://docs.gradle.com/build-scan-plugin">here</a>.</li>
<li>For more information on Gradle Enterprise, please see <a href="https://gradle.com/">here</a>.</li>
<li>For more information on build scan tags and values, please see <a href="/custom-data-in-build-scans">here</a>.</li>
</ul>
Gradle Kotlin DSL 1.02018-12-10T00:00:00-05:00https://blog.gradle.org/kotlin-dsl-1.0Paul Merlin
<p>The recently released <a href="https://docs.gradle.org/5.0/release-notes.html">Gradle 5.0</a> includes the <a href="https://github.com/gradle/kotlin-dsl/releases/tag/v1.0.2">Gradle Kotlin DSL v1.0</a> which is now ready for widespread use.</p>
<p>We want you to enjoy a build authoring experience with the benefits provided by Kotlin’s static type system in Intellij IDEA and Android Studio: auto-completion, smart content assist, quick access to documentation, navigation to source and context-aware refactoring.</p>
<video autoplay="" controls="" width="699" height="466">
<source src="/images/kotlin-dsl-1.0/kotlin-dsl-1.0-hi-res.mp4" type="video/mp4" />
<source src="/images/kotlin-dsl-1.0/kotlin-dsl-1.0-hi-res.webm" type="video/webm" />
</video>
<p>In case you missed it, you can watch Paul Merlin demonstrate these benefits in the <a href="https://www.youtube.com/watch?v=mAtrEPeAJSc">Type-safe build logic with Gradle Kotlin DSL</a> video from KotlinConf 2018.</p>
<p>If you prefer the flexibility and dynamic features of Groovy, that’s totally okay — the Groovy DSL will not be deprecated.
Using the Kotlin DSL today
You can get started quickly by following the <a href="https://guides.gradle.org/creating-new-gradle-builds/">Creating New Gradle Builds</a> guide. You can also follow one of the other guides like <a href="https://guides.gradle.org/building-kotlin-jvm-libraries/">Building Kotlin JVM Libraries</a> or <a href="https://guides.gradle.org/building-java-applications/">Building Java Applications</a>.</p>
<p>The <a href="https://docs.gradle.org/current/userguide/kotlin_dsl.html">Gradle Kotlin DSL Primer</a> user manual chapter is the best place to start learning more: it will cover all the basics and answer most of your questions.</p>
<p>If you are working with an existing build using the Gradle Groovy DSL you’ll be interested in the <a href="https://guides.gradle.org/migrating-build-logic-from-groovy-to-kotlin/">Migrating build logic from Groovy to Kotlin guide</a>.</p>
<p>Several community plugins now have Gradle Kotlin DSL snippets in their documentation: <a href="https://docs.spring.io/spring-boot/docs/current/gradle-plugin/reference/html/">Spring Boot</a>, <a href="https://bmuschko.github.io/gradle-docker-plugin/">Docker</a>, <a href="https://arturbosch.github.io/detekt/kotlindsl.html">Detekt</a>, <a href="https://github.com/jeremymailen/kotlinter-gradle">Kotlinter</a>, <a href="https://github.com/tbroyer/gradle-apt-plugin">APT</a>, <a href="https://github.com/mannodermaus/android-junit5/#readme">Android JUnit 5</a>, <a href="https://github.com/stoyicker/android-check-2">Android Check 2</a>, <a href="https://github.com/Triple-T/gradle-play-publisher">Android Play Publisher</a> and <a href="https://github.com/gradle/gradle/issues/6790">the list goes on</a>.</p>
<p>Last but not least, the <a href="https://docs.gradle.org/">Gradle documentation</a> contains build script excerpts that <a href="https://blog.gradle.org/groovy-kotlin-dsl-samples">demonstrate both the Groovy DSL and the Kotlin DSL</a>. This is the best place to find how to all things with each DSL; and it covers all Gradle features.</p>
<p>Again, a big thank you to all who helped make this a reality!</p>
<h2 id="feedback-wanted">Feedback wanted</h2>
<p>Community involvement has been instrumental to reach 1.0. It is also crucial going forward.</p>
<p>We want to know what you think works well or poorly in the Kotlin DSL. Feedback on the IDE experience, logging, and everything in between is welcome. Please discuss through the <a href="https://github.com/gradle/kotlin-dsl/issues">Kotlin DSL issue tracker</a> or the <code class="language-plaintext highlighter-rouge">#kotlin-dsl</code> channel on <a href="https://gradle-community.slack.com/">Gradle Community Slack</a> (which you can join using <a href="https://join.slack.com/t/gradle-community/shared_invite/enQtNDE3MzAwNjkxMzY0LTYwMTk0MWUwN2FiMzIzOWM3MzBjYjMxNWYzMDE1NGIwOTJkMTQ2NDEzOGM2OWIzNmU1ZTk5MjVhYjFhMTI3MmE">this link</a>).</p>
What’s new in Gradle 5.02018-12-04T00:00:00-05:00https://blog.gradle.org/what-is-new-in-gradle-5Eric Wendelin
<p>A lot has changed for Gradle since version 4.0 was released in mid-2017.</p>
<p>Gradle 5.0 is much faster and more memory efficient than previous versions of Gradle, has better tools to inspect and manage transitive dependencies, and can provide the IDE assistance you’d always hoped and expected through a new Kotlin DSL.</p>
<p>This 1-hour webcast demonstrates 4 key themes of Gradle 5.0:</p>
<p>What Gradle’s Kotlin DSL does and who should adopt it
How to take full advantage of incremental compilation and annotation processing
Why unexpected dependency version X was selected and how to fix it
What new task APIs are available in Gradle 5.0
If you prefer just the info, check out this page on the <a href="https://gradle.org/whats-new/gradle-5/">highlights of Gradle 5.0</a> and recent releases.</p>
<p style="text-align:center;">
<iframe width="560" height="315" src="https://www.youtube.com/embed/a9anXT8nM4I" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe></p>
Gradle Plugin Portal Approval Policy Update2018-10-18T00:00:00-04:00https://blog.gradle.org/new-plugin-portal-acceptance-criteriaEric Wendelin
<p>Gradle is updating the plugin approval policy for plugins submitted to the <a href="https://plugins.gradle.org">Gradle Plugin Portal</a>, effective today, to begin adding stronger security safeguards for plugin consumers.</p>
<p>First off, this does not affect plugins already on the plugin portal, just new plugins.
Gradle builds that use plugins will not be affected in any way.</p>
<h2 id="portal-acceptance-criteria-in-a-nutshell">Portal acceptance criteria in a nutshell</h2>
<p>Gradle will check the following for new plugins submitted to plugins.gradle.org:</p>
<ul>
<li>Description and project URL are valid and not misleading</li>
<li>The group ID and artifact ID are valid and not misleading</li>
</ul>
<p>In addition to these changes, plugins with a valid open-source repo URL will be prioritized over other plugins for approval.
Those that apply a SPDX-compatible license properly, even more so.</p>
<p>If your plugin doesn’t meet these requirements, we’ll let you know as soon as possible and give you a chance to re-submit when it does.
If your plugin cannot comply, please publish it to an alternative repository and <a href="https://docs.gradle.org/current/userguide/plugins.html#sec:plugin_management">use the <code class="language-plaintext highlighter-rouge">pluginManagement {}</code> DSL to configure where your <code class="language-plaintext highlighter-rouge">plugins {}</code> are resolved from</a>.</p>
<h2 id="what-happens-to-plugins-that-dont-adhere-to-this-criteria">What happens to plugins that don’t adhere to this criteria</h2>
<p>These checks do not yet apply for subsequent versions of a given plugin, and aren’t yet going to be retroactively enforced.</p>
<p>In the longer term, we will begin showing warnings on the plugin portal for plugins that don’t adhere to this policy, and may introduce additional automated checks to give plugin consumers information about plugins they’re viewing.</p>
<p>If you have questions or concerns, we encourage you to discuss in the <a href="https://discuss.gradle.org/c/help-discuss/plugin-portal">plugin-portal category on the Gradle forum</a>.
For any support requests, please <a href="https://github.com/gradle/plugin-portal-requests">open a GitHub issue</a>.</p>
How to show Groovy and Kotlin DSL samples side-by-side2018-10-11T00:00:00-04:00https://blog.gradle.org/groovy-kotlin-dsl-samplesEric Wendelin
<p>Gradle build script samples now have Kotlin DSL snippets alongside the Groovy snippets in <a href="https://docs.gradle.org/5.0/">Gradle 5.0 docs</a> and many <a href="https://gradle.org/guides/">Gradle guides</a>.</p>
<video controls="" autoplay="" muted="" preload="all" width="100%">
<source src="/images/groovy-kotlin-dsl-samples/multi-lang-sample-web.webm" type="video/webm" />
<source src="/images/groovy-kotlin-dsl-samples/multi-lang-sample-web.mp4" type="video/mp4" />
</video>
<p>We want you to be able to show both DSLs in your READMEs and web-based documentation, so this post shows ways you can display examples in <a href="#groovy-and-kotlin-dsl-samples-on-github">multiple languages on GitHub</a> and <a href="#groovy-and-kotlin-dsl-samples-on-the-web">on your websites</a>.</p>
<h2 id="groovy-and-kotlin-dsl-samples-on-github">Groovy and Kotlin DSL samples on GitHub</h2>
<p>It’s hard to find OSS projects that don’t have a <code class="language-plaintext highlighter-rouge">README.md</code> or <code class="language-plaintext highlighter-rouge">README.adoc</code>.
Though there aren’t many ways to make these README pages interactive, we can use the <code class="language-plaintext highlighter-rouge"><details></code> and <code class="language-plaintext highlighter-rouge"><summary></code> HTML elements within Markdown or Asciidoc achieve a bit of interactivity yet avoid the verbosity of having multiple subsequent samples <em>in view</em>.</p>
<p>Use these code snippets to show multi-language snippets on GitHub.</p>
<div class="exampleblock multi-language-sample" data-lang="markdown">
<div class="language-markdown highlighter-rouge"><pre class="highlight"><code data-lang="markdown"><span class="nt"><details</span> <span class="na">open</span><span class="nt">></span>
<span class="nt"><summary></span>Groovy<span class="nt"></summary></span>
<span class="p">```</span><span class="nl">groovy</span>
<span class="n">logging</span><span class="o">.</span><span class="na">captureStandardOutput</span> <span class="n">LogLevel</span><span class="o">.</span><span class="na">INFO</span>
<span class="n">println</span> <span class="s1">'A message which is logged at INFO level'</span>
<span class="p">```</span>
<span class="nt"></details></span>
<span class="nt"><details></span>
<span class="nt"><summary></span>Kotlin<span class="nt"></summary></span>
<span class="p">```</span><span class="nl">kotlin</span>
<span class="n">logging</span><span class="p">.</span><span class="n">captureStandardOutput</span><span class="p">(</span><span class="n">LogLevel</span><span class="p">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="n">println</span><span class="p">(</span><span class="s">"A message which is logged at INFO level"</span><span class="p">)</span>
<span class="p">```</span>
<span class="nt"></details></span>
</code></pre>
</div>
</div>
<div class="exampleblock multi-language-sample" data-lang="asciidoc">
<div class="language-asciidoc highlighter-rouge"><pre class="highlight"><code class="language-asciidoc" data-lang="asciidoc">++++
<details open>
<summary>Groovy</summary>
++++
[source,groovy]
----
logging.captureStandardOutput LogLevel.INFO
println 'A message which is logged at INFO level'
----
++++
</details>
++++
++++
<details>
<summary>Kotlin</summary>
++++
[source,kotlin]
----
logging.captureStandardOutput(LogLevel.INFO)
println("A message which is logged at INFO level")
----
++++
</details>
++++
</code></pre>
</div>
</div>
<p>Here it is in action:</p>
<video controls="" autoplay="" muted="" preload="all" width="100%">
<source src="/images/groovy-kotlin-dsl-samples/multi-lang-sample-demo-github.webm" type="video/webm" />
<source src="/images/groovy-kotlin-dsl-samples/multi-lang-sample-demo-github.mp4" type="video/mp4" />
</video>
<p>Note that this won’t work for <code class="language-plaintext highlighter-rouge">jekyll</code> or static websites, so there we have to write a bit of code.</p>
<h2 id="groovy-and-kotlin-dsl-samples-on-the-web">Groovy and Kotlin DSL samples on the web</h2>
<p>With a <a href="https://gist.github.com/eriwen/84d222972eeb5db3d5c784abdd7fbd5e" target="_blank">bit of CSS and JavaScript</a>, you can make a nice little language selector UI that remembers user selections.
The combination of this vanilla CSS, HTML, and JavaScript creates what you see at the top of the post.</p>
<p>A little bit about what the code does:</p>
<ul>
<li>First, the JS initializes the preferred language, getting and storing in <code class="language-plaintext highlighter-rouge">window.localStorage</code> so that user preferences are saved.</li>
<li>Next, it collapses all siblings that have <code class="language-plaintext highlighter-rouge">class=multi-language-sample</code> and generates tabs for each sample container element.</li>
<li>Finally, when a tab is clicked the JavaScript will measure and scroll the window so that the content doesn’t “jump” in the case where snippets are vastly longer or shorter — not bad, eh?</li>
</ul>
<p>The CSS is responsible for adds a nice Groovy or Kotlin logo to the appropriate tab, which is greyed out if the tab isn’t active.</p>
<p>Bonus: you can add <code class="language-plaintext highlighter-rouge">class="multi-language-text lang-foo"</code> to have any content be displayed only for the selected language.</p>
<p>I tested the code to be working in IE10+, but please let us know if there are bugs. I mean it’s not as fancy as the Google Cloud Storage docs, but I couldn’t find any good libraries to do this.</p>
<h2 id="conclusion">Conclusion</h2>
<p>These are a couple ideas you can use to show samples for multiple languages in your GitHub READMEs or on your sites.
I hope you find these useful for your projects.
Your other ideas and contributions are welcomed.</p>
Introducing Exemplar for Automated Samples Testing2018-10-09T00:00:00-04:00https://blog.gradle.org/documentation-samples-testing-exemplarEric Wendelin
<p>This post introduces a new library called <a href="https://github.com/gradle/exemplar">Exemplar</a>.
The goal of Exemplar is to ensure that users get outputs that you expect them to see.
It handles sample discovery, normalization (semantically equivalent results, perhaps from different environments), and flexible output verification.
It invokes any command-line tool in the environment to be invoked. You can also invoke <code class="language-plaintext highlighter-rouge">curl</code>, for example, to verify service API responses.</p>
<p>Gradle uses this library to verify examples in docs and guides, and remove boilerplate from integration tests.</p>
<p>Exemplar can be configured using a JUnit test runner (recommended) or using its APIs.
See examples below and in the <a href="https://github.com/gradle/exemplar">Exemplar GitHub repo</a>.</p>
<p><img src="/images/exemplar-information-flow.png" alt="Exemplar Information Flow" /></p>
<h2 id="use-cases-for-exemplar">Use cases for Exemplar</h2>
<p>It’s important that documentation samples are accurate.
Imagine the frustration, trying to learn something new and having the examples fail to work.</p>
<p>We think Exemplar works best as a documentation checking mechanism.
It is not meant to substitute other forms of integration testing however.</p>
<p>Exemplar is unique because it allows discovery of samples embedded in structured documents, as well as <code class="language-plaintext highlighter-rouge">OutputNormalizers</code> (bundled and custom ones), conveniences for using sample projects in “more traditional” integration tests (via <code class="language-plaintext highlighter-rouge">@UsesSample("path/to/sample")</code>), and allows samples to be programmatically modified (for extra environment setup, perhaps) before execution (<code class="language-plaintext highlighter-rouge">SampleModifier</code>).</p>
<p>Exemplar is focused on logging and exit code outputs, and verifying other side effects is cumbersome (checking created <em>files</em> may require extra commands).
Furthermore, although Exemplar can be used for any tool, it only has APIs for JVM projects.</p>
<h2 id="sample-project-testing-primer">Sample project testing primer</h2>
<p>The <code class="language-plaintext highlighter-rouge">sample-discovery</code> library in Exemplar will consider any directory containing a <code class="language-plaintext highlighter-rouge">*.sample.conf</code> file under the <em>samples root</em> as a sample project.</p>
<p>A sample config file is written in <a href="https://github.com/lightbend/config/blob/master/HOCON.md">HOCON</a> and tells <code class="language-plaintext highlighter-rouge">sample-check</code> how to run the sample project.
It can be simple:</p>
<div class="language-scss highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">executable</span><span class="p">:</span> <span class="n">echo</span>
<span class="n">args</span><span class="o">:</span> <span class="s2">"Hello World"</span>
</code></pre></div></div>
<p>… or more complex, with multiple steps:</p>
<div class="language-scss highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">commands</span><span class="p">:</span> <span class="p">[{</span>
<span class="na">executable</span><span class="p">:</span> <span class="n">gradle</span>
<span class="n">args</span><span class="o">:</span> <span class="n">originalInputs</span> <span class="n">incrementalReverse</span>
<span class="n">expected-output-file</span><span class="o">:</span> <span class="n">originalInputs</span><span class="o">.</span><span class="n">out</span>
<span class="n">allow-additional-output</span><span class="o">:</span> <span class="bp">true</span>
<span class="p">}</span><span class="o">,</span> <span class="p">{</span>
<span class="na">executable</span><span class="p">:</span> <span class="n">gradle</span>
<span class="n">args</span><span class="o">:</span> <span class="o">--</span><span class="n">quiet</span> <span class="n">removeOutput</span> <span class="n">incrementalReverse</span>
<span class="n">expected-output-file</span><span class="o">:</span> <span class="n">incrementalTaskRemovedOutput</span><span class="o">.</span><span class="n">out</span>
<span class="n">allow-disordered-output</span><span class="o">:</span> <span class="bp">true</span>
<span class="p">}</span><span class="o">]</span>
</code></pre></div></div>
<p><em>See the <a href="https://github.com/gradle/exemplar/blob/master/docs/README.adoc#external-sample-conf-reference">samples configuration reference</a> in the docs to learn what options are available.</em></p>
<p>A JUnit Test then declares which <code class="language-plaintext highlighter-rouge">@SampleRunner</code> and other aspects of samples tests.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@RunWith</span><span class="o">(</span><span class="nc">GradleSamplesRunner</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
<span class="nd">@SamplesRoot</span><span class="o">(</span><span class="s">"src/docs/samples"</span><span class="o">)</span>
<span class="nd">@SamplesOutputNormalizers</span><span class="o">({</span><span class="nc">JavaObjectSerializationOutputNormalizer</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="nc">FileSeparatorOutputNormalizer</span><span class="o">.</span><span class="na">class</span><span class="o">})</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">SamplesIntegrationTest</span> <span class="o">{}</span>
</code></pre></div></div>
<p>Executing this test will discover and run <em>external</em> (that is, not embedded) sample projects and generate a JUnit test report as you’d expect.</p>
<h2 id="embedded-samples-testing-primer">Embedded samples testing primer</h2>
<p>Exemplar can discover samples within <a href="https://asciidoctor.org/">Asciidoctor</a> source files using Asciidoctor’s AST.
Here’s an example that uses <code class="language-plaintext highlighter-rouge">SampleModifiers</code> as well to setup Samples’ environments as well.</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.Sample title
====
[.testable-sample] (1)
=====
.hello.rb (2)
[source,ruby] (3)
-----
puts "hello, #{ARGV[0]}" (4)
-----
[.sample-command] (5)
-----
$ ruby hello.rb world (6)
hello, world (7)
-----
=====
====
</code></pre></div></div>
<p><em>Please read the <a href="https://github.com/gradle/exemplar/blob/master/docs/README.adoc#configuring-embedded-samples">docs for embedded samples</a> to learn about the Asciidoctor elements required to allow this to work.</em></p>
<p>We run this test using the <code class="language-plaintext highlighter-rouge">EmbeddedSamplesRunner</code> pointing to the <code class="language-plaintext highlighter-rouge">docs/</code> root directory where Exemplar will look for documentation files.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@RunWith</span><span class="o">(</span><span class="nc">EmbeddedSamplesRunner</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
<span class="nd">@SamplesRoot</span><span class="o">(</span><span class="s">"docs"</span><span class="o">)</span>
<span class="nd">@SampleModifiers</span><span class="o">({</span><span class="nc">SetupEnvironmentSampleModifier</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="nc">ExtraCommandArgumentsSampleModifier</span><span class="o">.</span><span class="na">class</span><span class="o">})</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">EmbeddedSamplesIntegrationTest</span> <span class="o">{}</span>
</code></pre></div></div>
<p>Executing this test will extract the code from Asciidoctor into a temporary project, run it, and verify it matches the declared output in the doc.</p>
<h2 id="adopt-and-contribute">Adopt and contribute</h2>
<p>The best way to get started is to read the Exemplar docs and check out <a href="https://github.com/gradle/exemplar">Exemplar’s samples</a> and <a href="https://github.com/gradle/exemplar/tree/master/sample-check/src/test/java/org/gradle/samples/test/runner">sample tests</a>.</p>
<p>Exemplar is at version 0.6 at the time of writing this post, which means that the APIs may change before v1.0.
That said, given that the API has gone through some revisions already and has been adopted by Gradle, breaking changes are unlikely before version 1.0.</p>
<p>Here are some of the things that might be good expansions of the library, and I’d love your help if you’re eager to file ideas or submit pull requests to the <a href="https://github.com/gradle/exemplar">gradle/exemplar GitHub repo</a>.</p>
<ul>
<li>More standard normalizers such as date, time, or duration normalization</li>
<li>Generate structured metadata for search indexing</li>
<li>Allow expected output to be inlined in sample config files</li>
<li>Samples discovery embedded in Markdown or other documentation formats (#2938)</li>
</ul>
<p>Please reach out to me (<a href="https://twitter.com/eriwen">@eriwen on Twitter</a>) if you’d like guidance adopting or contributing to this project.</p>
Introducing source dependencies2018-09-25T00:00:00-04:00https://blog.gradle.org/introducing-source-dependenciesAdam Murdoch
<p>This post introduces a new Gradle dependency management feature called “source dependencies”.</p>
<p>Normally, when you declare a dependency on a library, Gradle looks for the library’s binaries in a binary repository, such as JCenter or Maven Central, and downloads the binaries for use in the build.</p>
<p>Source dependencies allow you to instead have Gradle automatically check out the source for the library from Git and build the binaries locally on your machine, rather than downloading them.</p>
<p>We’d love to get your feedback on this feature. Please try it out and let us know what works for
you and any problems you run into. You can leave feedback on the <a href="https://discuss.gradle.org/">Gradle forums</a> or raise issues on the <a href="https://github.com/gradle/gradle-native">Gradle native</a> GitHub repository.</p>
<p>You should be aware that this feature is currently incubating and may change in breaking ways in future releases of Gradle. Please do not use this feature in production until the feature is promoted to stable in a future release.</p>
<h2 id="how-can-you-try-it">How can you try it?</h2>
<p>You can find many samples that use source dependencies in the <a href="https://github.com/gradle/native-samples">native-samples</a> repository. C++ builds are a natural place to use source dependencies, but this isn’t a C++ specific feature. Source dependencies work for any type of build that use Gradle’s dependency management features, such as Java, Kotlin, or Android builds. You can try this feature out with the latest <a href="https://gradle.org/releases/">Gradle 4.10</a> or a <a href="https://gradle.org/nightly/">nightly</a>.</p>
<p>Let’s look at an <a href="https://github.com/gradle/native-samples#application-with-source-library-dependencies-source-dependencies">example application</a>. This sample shows an application that uses a library that lives in a separate Git repository.</p>
<p>A source dependency looks the same as a binary dependency in the build file. You declare a dependency on a particular version of a <a href="https://docs.gradle.org/current/userguide/dependency_management_terminology.html#sub:terminology_module">module</a>, such as a library. Here, our sample needs version 1.0 of the “utilities” library:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dependencies</span> <span class="o">{</span>
<span class="n">implementation</span> <span class="s1">'org.gradle.cpp-samples:utilities:1.0'</span>
<span class="o">}</span>
</code></pre></div></div>
<p>You also need to tell Gradle where to find the source for the library. This is similar to how you need to tell Gradle where to find binaries by defining some binary repositories in your build, however, the syntax is different. In the <code class="language-plaintext highlighter-rouge">settings.gradle</code> file we define a source mapping:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sourceControl</span> <span class="o">{</span>
<span class="n">gitRepository</span><span class="o">(</span><span class="s2">"https://github.com/gradle/native-samples-cpp-library.git"</span><span class="o">)</span> <span class="o">{</span>
<span class="n">producesModule</span><span class="o">(</span><span class="s2">"org.gradle.cpp-samples:utilities"</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Now, when Gradle needs to find a version of the “utilities” library, it will look for a matching tag in the Git repository. Gradle will check out the matching Git tag, build the binaries and make the result available.
This works the same way as an <a href="https://docs.gradle.org/current/userguide/composite_builds.html">included build</a>.</p>
<p>Here’s the result of running <code class="language-plaintext highlighter-rouge">gradle assemble</code> on this sample:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> Task :native-samples-cpp-library:list:compileDebugCpp
> Task :native-samples-cpp-library:utilities:compileDebugCpp
> Task :native-samples-cpp-library:list:linkDebug
> Task :compileDebugCpp
> Task :native-samples-cpp-library:utilities:linkDebug
> Task :linkDebug
> Task :installDebug
> Task :assemble
BUILD SUCCESSFUL in 4s
3 actionable tasks: 3 executed
</code></pre></div></div>
<p>Gradle has cloned the source of the libraries, built the library binaries and then linked the application against these.</p>
<p>You can also declare a dependency on a branch, using a slightly different syntax. This <a href="https://github.com/gradle/native-samples#application-with-dependency-on-upstream-branch-dependency-on-upstream-branch">example application</a> demonstrates how to do this.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dependencies</span> <span class="o">{</span>
<span class="n">implementation</span><span class="o">(</span><span class="s1">'org.gradle.cpp-samples:utilities'</span><span class="o">)</span> <span class="o">{</span>
<span class="n">version</span> <span class="o">{</span>
<span class="n">branch</span> <span class="o">=</span> <span class="s1">'release'</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Source dependencies work for plugins too, but currently need some additional configuration in the <code class="language-plaintext highlighter-rouge">settings.gradle</code> in order to work. This <a href="https://github.com/gradle/native-samples#application-uses-a-library-built-by-cmake-cmake-library">example library</a> shows how to do this.</p>
<h2 id="what-are-source-dependencies-useful-for">What are source dependencies useful for?</h2>
<p>Source dependencies are useful when the binaries for a module that you use are not available in a binary repository. For example:</p>
<ul>
<li>When you’re working off a branch of a library and binaries have not yet been published for this branch. For example, you may want to use some bug fix from a branch that has not yet been merged or released.</li>
<li>When you’re using a library that publishes source and no binaries. This is common for C++ libraries - and applications - where the libraries are released in source form only and you have to build your own binaries.</li>
<li>When you’re using a native library that does not publish binaries for your target platform. It is common for a C++ library or application to publish binaries only for certain common operating systems or tool chains. To use the library on other platforms, you have to build your own binaries.</li>
<li>When you don’t want to have to wait for another system, such as a CI server, to publish binaries for a dependency. For example, you might want to experiment with changes to several libraries.</li>
</ul>
<p>Source dependencies make these use cases simpler to implement. Gradle takes care of automatically checking out the correct versions of dependencies, making sure the binaries are built when required. It does this everywhere that the build is run. The checked out project doesn’t even need to have an existing Gradle build. <a href="https://github.com/gradle/native-samples#application-uses-libraries-that-are-not-built-by-gradle-injected-plugins">This example</a> shows a Gradle build consuming two source dependencies that have <em>no</em> build system by injecting a Gradle build via plugins. The injected configuration could do anything a regular Gradle plugin can do, such as wrapping an existing CMake or Maven build.</p>
<p>Source dependencies can be combined with the Gradle <a href="https://guides.gradle.org/using-build-cache/">build cache</a>, to make these use cases efficient when everyone on your team is building the same dependencies from source. The Gradle build cache ensures that once the binaries have been built, they are reused everywhere else.</p>
<p>Source dependencies can be used alongside binary dependencies, and you can mix and match so that some dependencies are built from source and others are downloaded as binaries. Source dependencies also work nicely with <a href="https://docs.gradle.org/current/userguide/composite_builds.html">composite builds</a>.</p>
<h2 id="how-is-this-different-to-a-composite-build">How is this different to a composite build?</h2>
<p>Mostly, it’s not. Source dependencies are based on Gradle’s <a href="https://docs.gradle.org/current/userguide/composite_builds.html">composite build</a> feature and add automatic provisioning of the source code, to help with some specific use cases.</p>
<p>Source dependencies are different to included builds in <em>intent</em>. An included build represents some module that you are currently working on. Dependency resolution will always use binaries from the included build regardless of which version of the module is requested in the build, so that you can see the effect of your changes.</p>
<p>A source dependency, on the other hand, represents some module that you use, but that just happens to be available in source form rather than binary form. Dependency resolution will always select a version of the source dependency that matches whatever is requested in the build. With an included build, you can modify sources and directly see the changes, but with source dependencies, you need to commit and push changes before you can see their effect.</p>
<h2 id="missing-features">Missing features</h2>
<p>Source dependency support is very much experimental and is missing some important features:</p>
<ul>
<li><a href="https://scans.gradle.com/">Build scans</a> do not yet support builds that use source dependencies.</li>
<li>IDE support is mostly not yet available. Most IDE integrations will fail with an error when source dependencies are used by the build. Source dependencies are currently only supported by Gradle’s Xcode integration for C++.</li>
<li><a href="https://docs.gradle.org/current/userguide/dependency_locking.html">Dependency locking</a> does not support source dependencies.</li>
<li>Source dependencies also have the same <a href="https://docs.gradle.org/current/userguide/composite_builds.html#included_build_substitution_limitations">restrictions</a> as included builds.</li>
<li>You cannot have a source dependency on a build that contains included builds. However, you can have a source dependency on a build that uses source dependencies.</li>
<li>You cannot yet mix source and binary dependencies for a particular module. Each module must come either from a Git repository or from a binary repository, but not both.</li>
<li>Only Git repositories are supported.</li>
</ul>
<h2 id="roadmap">Roadmap</h2>
<p>We’d like to get your feedback and based on this implement the missing features listed above. Once we’re happy that the DSL and APIs are useful and behave well, we will promote the feature to stable. Let us know what you think on the <a href="https://discuss.gradle.org/">Gradle forums</a> or the <a href="https://github.com/gradle/gradle-native">Gradle native</a> GitHub repository.</p>
Stop rerunning your tests2018-09-08T00:00:00-04:00https://blog.gradle.org/stop-rerunning-testsStefan Oehme
<p>Tests are usually the longest running operation in your development process.
Running them unnecessarily is the ultimate time waster.
Gradle helps you avoid this cost with its <a href="https://docs.gradle.org/current/userguide/build_cache.html">build cache</a>
and <a href="https://docs.gradle.org/current/userguide/more_about_tasks.html#sec:up_to_date_checks">incremental build</a> features.
It knows when any of your test inputs, like your code, your dependencies
or system properties, have changed. If everything stays the same, Gradle will
skip the test run, saving you a lot of time.</p>
<p>So you can imagine my desperation when I see snippets like this on StackOverflow:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tasks</span><span class="o">.</span><span class="na">withType</span><span class="o">(</span><span class="n">Test</span><span class="o">)</span> <span class="o">{</span>
<span class="n">outputs</span><span class="o">.</span><span class="na">upToDateWhen</span> <span class="o">{</span> <span class="kc">false</span> <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Let’s talk about what this means and why it is a bad idea.</p>
<h2 id="communicating-intent">Communicating intent</h2>
<p>The above snippet just says “Never reuse this test’s output”.
But why? Is it because there is some hidden input that Gradle doesn’t know about?
Or is it because the test produces random outputs? The reader can’t tell.</p>
<h2 id="deterministic-tests-dont-need-reruns">Deterministic tests don’t need reruns</h2>
<blockquote>
<p>“Insanity is doing the same thing over and over and expecting different results”</p>
<p>- Not Albert Einstein</p>
</blockquote>
<p>The vast majority of your tests should be deterministic, i.e. given the same inputs they should produce the same result.
If this is not the case, your project is in serious trouble. Stop reading this post and start fixing your code!</p>
<p>Rerunning deterministic tests is a waste of your team’s time.</p>
<h2 id="non-deterministic-tests">Non-deterministic tests</h2>
<p>There are a few reasons why you might want to rerun <em>some</em> tests in <em>some</em> cases even though none of the code changed.
In these cases, you should model the additional input properly.
Tell Gradle what exactly makes your tests non-deterministic.</p>
<h3 id="randomized-tests">Randomized tests</h3>
<p>Some tests use randomization to improve the quality of your software.</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Random_testing">Random testing</a> can be used to ensure that the production code can handle all kinds of inputs,
not just the ones that the developer came up with.</li>
<li><a href="https://en.wikipedia.org/wiki/Mutation_testing">Mutation testing</a> changes the production code in subtle way (e.g. introducing off-by-one errors)
and checks whether your test suite catches these mistakes.</li>
</ul>
<p>Make this randomization explicit by making the random seed an input to your task:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">task</span> <span class="nf">randomizedTest</span><span class="o">(</span><span class="nl">type:</span> <span class="n">Test</span><span class="o">)</span> <span class="o">{</span>
<span class="n">systemProperty</span> <span class="s2">"random.testing.seed"</span><span class="o">,</span> <span class="k">new</span> <span class="n">Random</span><span class="o">().</span><span class="na">nextInt</span><span class="o">()</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This will force Gradle to always rerun that test, because it will always have a different seed.
Even better, you could make the seed user-configurable to locally reproduce bugs that were found on your build server.</p>
<h3 id="system-integration-tests">System integration tests</h3>
<p>System integration tests verify your application against realistic versions of other applications that are controlled by other teams.
They may fail even if you didn’t change anything, e.g. because another team broke an API.
They also tend to be among the slowest tests in your code base, so you don’t want to rerun them just because you changed some documentation.
A good compromise may be to check integration at least once a day, even if nothing on your side has changed.</p>
<p>Make this interval part of your test inputs:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">task</span> <span class="nf">systemIntegrationTest</span><span class="o">(</span><span class="nl">type:</span> <span class="n">Test</span><span class="o">)</span> <span class="o">{</span>
<span class="n">inputs</span><span class="o">.</span><span class="na">property</span> <span class="s2">"integration.date"</span><span class="o">,</span> <span class="n">LocalDate</span><span class="o">.</span><span class="na">now</span><span class="o">()</span>
<span class="o">}</span>
</code></pre></div></div>
<p>You can then set up an automated build that runs this test in the morning before everyone starts working and let it push the result to a shared build cache.
When your team comes to work, the test result will be downloaded from the cache and the tests won’t have to rerun for that day.
They can then use the saved time on more productive tasks like fixing bugs or developing new features.</p>
<h3 id="flaky-tests">Flaky tests</h3>
<p>Sometimes you encounter a bug that only makes a test fail 1 out of 10 times.
In order to analyze the situation, you want Gradle to rerun the test even if it was successful before.
In this case it is reasonable to use the random input approach I showed above.
However, I find it more productive to wrap the flaky test in an endless loop.
This way I can keep the debugger in the IDE running and even make small changes on the fly without having to restart.</p>
<h2 id="model-your-tests-properly">Model your tests properly</h2>
<p>Properly modeling your requirements is just as important for your build logic as it is for your production code.
Doing it right will make your builds faster and your team more productive.</p>
<p>The <code class="language-plaintext highlighter-rouge">java</code> plugin’s built-in <a href="https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_test"><code class="language-plaintext highlighter-rouge">test</code></a> task is meant for deterministic and fast unit tests.
It may seem convenient to put all your tests into this one task and rerun them on every build, because a few of them are non-deterministic.
This lazy approach will save you a few minutes of thinking and coding, but the long term costs are huge, slowing everyone on your team down every day.</p>
<p>Instead, create additional <a href="https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html"><code class="language-plaintext highlighter-rouge">Test</code></a> tasks for different types of tests like functional, performance, or randomized tests. For each one of them, consider when it needs to rerun and model that as an input to the task.</p>
<p>Don’t waste your time.</p>
Get ready for Kotlin DSL 1.02018-08-26T00:00:00-04:00https://blog.gradle.org/gradle-kotlin-dsl-release-candidateEric Wendelin
<p><a href="https://github.com/gradle/kotlin-dsl/releases/tag/v1.0-RC3">Gradle Kotlin DSL 1.0 release candidate</a> is generally available, included in Gradle 4.10. The Kotlin DSL is nearly ready for widespread use.</p>
<p>We want you to enjoy a build authoring experience with the benefits provided by Kotlin’s static type system: context-aware refactoring, smart content assist, debuggable build scripts, and quick access to documentation.
In case you haven’t seen it, you can watch Rodrigo B. de Oliveira demonstrate these benefits in <a href="https://youtu.be/lg5hPBEIQ2E?t=27m33s">this KotlinConf 2017 video</a>.</p>
<p>Kotlin DSL 1.0 final will be released with Gradle 5.0, which is the <em>next version of Gradle</em>.
After version 1.0, the Kotlin DSL will not introduce any more breaking changes without a deprecation cycle.</p>
<p>Please try the Kotlin DSL and submit feedback. Guidance for doing that follows.</p>
<h2 id="trying-the-kotlin-dsl-today">Trying the Kotlin DSL today</h2>
<p>The <em><a href="https://guides.gradle.org/migrating-build-logic-from-groovy-to-kotlin/">Migrating build logic from Groovy to Kotlin</a></em> guide is the best place to start: it will cover all the basics and answer most of your questions.</p>
<p>You should also consider these resources for use cases not covered by the migration guide:</p>
<ul>
<li>Dozens of <a href="https://github.com/gradle/kotlin-dsl/tree/master/samples">Kotlin DSL samples</a> are available through GitHub</li>
<li>Some of the <a href="https://guides.gradle.org">guides</a> and <a href="https://docs.gradle.org/nightly/">nightly Gradle docs</a> have Kotlin DSL samples. (Interested in contributing? Follow <a href="https://github.com/gradle/gradle/issues/6442">these instructions</a>).</li>
</ul>
<h2 id="feedback-wanted">Feedback wanted</h2>
<p>Your feedback is crucial for this project to reach this milestone.</p>
<p>We want to know your experiences, good and bad, authoring the Kotlin DSL, understanding the logging, with your IDE, and everywhere in between.
Please discuss through <a href="https://github.com/gradle/kotlin-dsl/issues">Kotlin DSL issue tracker</a> or through the <code class="language-plaintext highlighter-rouge">#kotlin-dsl</code> channel on <a href="https://gradle-community.slack.com/">Gradle Community Slack</a> (which you can join using <a href="https://join.slack.com/t/gradle-community/shared_invite/enQtNDE3MzAwNjkxMzY0LTYwMTk0MWUwN2FiMzIzOWM3MzBjYjMxNWYzMDE1NGIwOTJkMTQ2NDEzOGM2OWIzNmU1ZTk5MjVhYjFhMTI3MmE">this link</a>).</p>
<p>We would not be where we are today without you.
Special thanks to <a href="https://github.com/JLLeitschuh">Jonathan Leitschuh</a>, <a href="https://github.com/jnizet">Jean-Baptiste Nizet</a>, <a href="https://github.com/ligee">Ilya Chernikov</a>, <a href="https://github.com/StefMa">Stefan M.</a>, <a href="https://github.com/mkobit">Mike Kobit</a>, the awesome Kotlin team, and dozens of others who’ve helped along the way.</p>
<p><em>Onward to Gradle Kotlin DSL 1.0!</em></p>
Evolving the Gradle API to reduce configuration time2018-07-09T00:00:00-04:00https://blog.gradle.org/preview-avoiding-task-configuration-timeSterling Greene
<p>This post introduces a new API for declaring and configuring <a href="https://docs.gradle.org/current/userguide/custom_tasks.html">Gradle Tasks</a> in build scripts and plugins. We intend for this new API to eventually replace the existing API because it allows Gradle to avoid configuring unnecessary build logic. Use of the new API will become the default recommendation soon, but the existing API will go through our usual deprecation process over several major releases.</p>
<p>We are asking early adopters to try out the new Gradle Tasks API to iron out any issues and gather feedback. We’ve created a <a href="https://docs.gradle.org/4.9/userguide/task_configuration_avoidance.html">new user manual chapter</a> to give a quick introduction to the feature and explains some guidelines for migrating your build to use the new API.</p>
<p>We welcome any feedback you may have about this API. You can leave feedback on this <a href="https://github.com/gradle/gradle/issues/5664">Gradle issue</a>.</p>
<h2 id="preview-of-new-gradle-tasks-api-in-gradle-49">Preview of new Gradle Tasks API in Gradle 4.9</h2>
<p>One of the major differences between the existing and new Gradle Tasks API is whether or not Gradle spends the time to create <code class="language-plaintext highlighter-rouge">Task</code> instances and run configuration code. The new API allows Gradle to delay or completely avoid configuring tasks that will never be executed in a build. For example, when compiling code, Gradle does not need to configure tasks that run tests.</p>
<p>The time spent creating and configuring tasks that are never used is a contributor to overall Gradle configuration time. Configuration time affects every build, so making it faster is good for everyone.</p>
<p>The existing API also eagerly creates <code class="language-plaintext highlighter-rouge">Task</code> instances as soon as possible. This can make the ordering of plugins more fragile. The new API is intended to make the relationship between a <code class="language-plaintext highlighter-rouge">Task</code> and something that needs it more explicit by passing around a <a href="https://docs.gradle.org/current/userguide/lazy_configuration.html"><code class="language-plaintext highlighter-rouge">Provider</code></a> for a <code class="language-plaintext highlighter-rouge">Task</code>.</p>
<p>We have also seen in some builds that configuring particular tasks are very expensive because they reach out to webservices, they run <code class="language-plaintext highlighter-rouge">git status</code> or they need to parse something. If those tasks are rarely used, every build is paying the price to configure that task. The new API allows plugin or build authors to declare the expensive tasks so that they’re only configured when necessary.</p>
<p>Later Gradle releases will provide more details and samples for the new API once it becomes the default recommendation.</p>
<h3 id="measured-impact-to-gradle-builds">Measured impact to Gradle builds</h3>
<p>For the last few months, we have been dogfooding the new Gradle Tasks API in the <a href="https://github.com/gradle/gradle">gradle/gradle</a> build, <a href="https://gradle.com/">Gradle Enterprise</a> and a typical <a href="https://github.com/gradle/perf-enterprise-large">large project</a>.</p>
<p><img src="/images/task-configuration-avoidance/gradle-help-chart.png" alt="Making the gradle/gradle build faster" /></p>
<p>Over that time, we’ve made several changes to reduce configuration time in the Gradle build <a href="https://builds.gradle.org/repository/download/Gradle_Check_PerformanceTestCoordinator/13921862:id/report-performance-performance-tests.zip%21/report/tests/help-on-the-gradle-build-comparing-gradle.html">from 1.7s to 1.4s</a> on our Linux performance agents. Part of these improvements come from the new API, the Gradle build still configures several hundred tasks unnecessarily, so we believe that we can reduce this number further.</p>
<p>Below, we’ve plotted the improvements directly related to the new API. We’ve measured this on a mid-2014 MacBook Pro (2.6 GHz Intel Core i5, 16GB RAM) with several different projects.</p>
<p><img src="/images/task-configuration-avoidance/gradle-eager-lazy.png" alt="Improving other builds" /></p>
<p>The perf-enterprise-large project is a generated project we use in our performance tests that has nearly 350 Java modules. When configuring this build with the existing API, Gradle creates and configures over 10000 tasks. With the new API, Gradle only configures 349 (97% of all tasks are avoided). Changes in a future version of Gradle will make this 1 task. From the graph, we can see that the average configuration time goes from 936ms to 703ms (-233ms).</p>
<p>The second group of plots is for the gradle/gradle build. Without the new API, Gradle would create and configure over 6300 tasks. We have more work to do, but we avoid creating and configuring 90% of all tasks. Our average configuration time dropped from 1325ms to 1117ms (-208ms).</p>
<p>For our closed-source Gradle Enterprise project, we’ve only just started converting it to the new API. We only avoid about 64% of all tasks. Our average configuration time dropped from 1636ms to 1467ms (-169ms).</p>
<p>We expect other builds to see a similar reduction when many tasks are avoided when they are not needed. Some builds may see a larger impact if they are much larger (hundreds of subprojects) or configure very expensive tasks. We’re working with prominent plugin authors to use the new API, so most builds will see the task avoidance benefits.</p>
<h3 id="why-a-new-api">Why a new API?</h3>
<p>Unfortunately, the existing API could not be adapted to meet our new requirements:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Task</code> instances should not be created immediately. Many existing APIs return <code class="language-plaintext highlighter-rouge">Task</code> and we could not break binary compatibility.</li>
<li>Tasks should not be created or configured unless needed. Many existing APIs <em>could</em> be adapted to delay creation or configuration, but this would have silently broken many builds in hard to diagnose ways.</li>
</ul>
<p>The new API is designed to work along side the existing API, so tasks created with the existing API are visible to the new API and vice versa. Builds can gradually migrate to the new API, but mixing the existing API will negate some of the benefits of the new API. Any existing API that requires a <code class="language-plaintext highlighter-rouge">Task</code> instance will force tasks created by the new API to be created as if they were created via the existing API.</p>
<p>The new API is also intended to be similar enough to the existing API that migration is easy.</p>
<h3 id="migrating-from-the-existing-gradle-tasks-api-to-the-new-api">Migrating from the existing Gradle Tasks API to the new API</h3>
<p>Our <a href="https://docs.gradle.org/4.9/userguide/task_configuration_avoidance.html#sec:old_vs_new_configuration_api_overview">user manual chapter</a> provides a reference for mapping existing API to new API.</p>
<p>As a quick and dirty reference:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">tasks.create(...)</code> becomes <code class="language-plaintext highlighter-rouge">tasks.register(...)</code></li>
<li><code class="language-plaintext highlighter-rouge">tasks.withType(SomeType) { }</code> becomes <code class="language-plaintext highlighter-rouge">tasks.withType(SomeType).configureEach { }</code></li>
<li><code class="language-plaintext highlighter-rouge">tasks.all { }</code> becomes <code class="language-plaintext highlighter-rouge">tasks.configureEach { }</code></li>
<li><code class="language-plaintext highlighter-rouge">tasks.getByName(...)</code> becomes <code class="language-plaintext highlighter-rouge">tasks.named(...)</code></li>
<li>For the Groovy Gradle DSL, there is not a replacement for <code class="language-plaintext highlighter-rouge">task foo(...) { }</code> that uses the new API.</li>
</ul>
<p>Please keep in mind that configuration blocks that are used with the new API are not guaranteed to execute all of the time.</p>
<p>In the existing API, if you were to use <code class="language-plaintext highlighter-rouge">tasks.withType(...)</code>:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s2">"java"</span>
<span class="o">}</span>
<span class="n">tasks</span><span class="o">.</span><span class="na">withType</span><span class="o">(</span><span class="n">JavaCompile</span><span class="o">)</span> <span class="o">{</span> <span class="n">javaCompile</span> <span class="o">-></span>
<span class="n">println</span> <span class="s2">"Hello, "</span> <span class="o">+</span> <span class="n">javaCompile</span><span class="o">.</span><span class="na">name</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Running <code class="language-plaintext highlighter-rouge">gradle help</code> or <code class="language-plaintext highlighter-rouge">gradle build</code> or <code class="language-plaintext highlighter-rouge">gradle compileJava</code> would log “Hello, compileJava” and “Hello, compileTestJava”. The tasks that are configured do not change depending on what needs to be executed.</p>
<p>If we instead use the new API <code class="language-plaintext highlighter-rouge">tasks.withType(...).configureEach</code>:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s2">"java"</span>
<span class="o">}</span>
<span class="n">tasks</span><span class="o">.</span><span class="na">withType</span><span class="o">(</span><span class="n">JavaCompile</span><span class="o">).</span><span class="na">configureEach</span> <span class="o">{</span> <span class="n">javaCompile</span> <span class="o">-></span>
<span class="n">println</span> <span class="s2">"Hello, "</span> <span class="o">+</span> <span class="n">javaCompile</span><span class="o">.</span><span class="na">name</span>
<span class="o">}</span>
</code></pre></div></div>
<p>We see the behavior is very different. Gradle configures more or less of the build depending on what is required for execution.</p>
<ul>
<li>Running <code class="language-plaintext highlighter-rouge">gradle help</code> will not log any “Hello” messages. Gradle is able to avoid configuring the <code class="language-plaintext highlighter-rouge">JavaCompile</code> tasks because they are never executed.</li>
<li>Running <code class="language-plaintext highlighter-rouge">gradle build</code> <em>will</em> log “Hello, compileJava” and “Hello, compileTestJava”. Gradle has to configure the <code class="language-plaintext highlighter-rouge">JavaCompile</code> tasks because they’re both executed.</li>
<li>Running <code class="language-plaintext highlighter-rouge">gradle compileJava</code> will log <em>only</em> “Hello, compileJava”. Gradle only has to configure the <code class="language-plaintext highlighter-rouge">compileJava</code> task because it’s executed. Task <code class="language-plaintext highlighter-rouge">compileTestJava</code> is not executed, so it is not created or configured.</li>
</ul>
<h2 id="your-feedback-wanted">Your Feedback Wanted</h2>
<p>Included in <a href="https://gradle.com/enterprise/releases/2018.3">Gradle Enterprise 2018.3</a>, you can see the <a href="https://gradle.com/enterprise/releases/2018.3#reduce-configuration-time-by-leveraging-task-creation-avoidance">progress of using the new task API</a> in your build scans. For many of the built-in Gradle plugins, we have switched to using the new APIs. For builds using Java, you may see a large number of tasks that are never configured already.</p>
<p>This Gradle Tasks API is a work-in-progress, but we believe that it is complete enough to replace the existing API in all scenarios. We plan to incorporate any feedback into the next release, which will contain more documentation and examples for using the new API. Our intention is to make this API production ready as soon as possible.</p>
<p>Please <a href="https://docs.gradle.org/4.9/userguide/task_configuration_avoidance.html#sec:what_do_we_want">try this API out in your plugins and builds</a> and let us know what you think. We’ve included a <a href="https://docs.gradle.org/4.9/userguide/task_configuration_avoidance.html#sec:new_task_gradle_profiler_scenario">section in the user manual</a> to collect data for a performance comparisons between the existing and new APIs. We would love to see your results and hear what you think works well, what’s confusing, and what is missing that would block you from using this new API over the existing one. You can leave feedback on this <a href="https://github.com/gradle/gradle/issues/5664">Gradle issue</a>.</p>
Fixing Gradle dependency resolution when TLS v1.1 and v1.0 support is discontinued2018-06-21T00:00:00-04:00https://blog.gradle.org/unable-to-download-maven-central-bintrayEric Wendelin
<p>Maven Central and Bintray have announced that they will discontinue support for TLS v1.1 and below. Here’s what you need to know to correct your Gradle builds if you’re affected.</p>
<p><em>You will need to take action if you are using Java 6 or 7 and using Gradle versions 2.1 through 4.8.</em></p>
<h3 id="how-to-check-if-youre-affected">How to check if you’re affected</h3>
<p>You may already be getting one of the following errors from your build after an error message saying: “Could not resolve [coordinates]”:</p>
<blockquote>
<p>Received fatal alert: protocol_version</p>
</blockquote>
<p>or</p>
<blockquote>
<p>peer not authenticated</p>
</blockquote>
<!--break-->
<p>If not, you can check whether you will be affected by running the following:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gradle <span class="nt">--version</span> <span class="c"># Without Gradle Wrapper</span>
./gradlew <span class="nt">--version</span> <span class="c"># Using Gradle Wrapper on *nix</span>
gradlew.bat <span class="nt">--version</span> <span class="c"># Using Gradle Wrapper on Windows</span>
</code></pre></div></div>
<p>it will print something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>------------------------------------------------------------
Gradle 3.5
------------------------------------------------------------
Build time: 2017-04-10 13:37:25 UTC
Revision: b762622a185d59ce0cfc9cbc6ab5dd22469e18a6
Groovy: 2.4.10
Ant: Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM: 1.7.0_80 (Oracle Corporation 24.80-b11)
OS: Mac OS X 10.13.5 x86_64
</code></pre></div></div>
<p>You must take action if all of these are true:</p>
<ul>
<li>JVM version is Java 7u130 or lower</li>
<li><strong>and</strong> Gradle version is between 2.1 and 4.8, inclusive</li>
<li><strong>and</strong> you have declared a <code class="language-plaintext highlighter-rouge">repository {}</code> of <code class="language-plaintext highlighter-rouge">mavenCentral()</code> or <code class="language-plaintext highlighter-rouge">jcenter()</code></li>
</ul>
<h3 id="how-to-use-tls-12-for-dependency-resolution">How to use TLS 1.2 for dependency resolution</h3>
<p>You can take any one of the following actions to use TLS v1.2+:</p>
<ul>
<li>Run Gradle with Java 1.7.0_131-b31 or above</li>
<li><strong>or</strong> upgrade to <a href="https://github.com/gradle/gradle/releases/tag/v4.8.1">Gradle 4.8.1</a> or above</li>
<li><strong>or</strong> replace <code class="language-plaintext highlighter-rouge">mavenCentral()</code> with <code class="language-plaintext highlighter-rouge">maven { url = "http://repo.maven.apache.org/maven2" }</code> and <code class="language-plaintext highlighter-rouge">jcenter()</code> with <code class="language-plaintext highlighter-rouge">maven { url = "http://jcenter.bintray.com" }</code></li>
</ul>
<p>The first two solutions are recommended, as the third opens a possible attack vector.</p>
<h3 id="other-resources">Other resources</h3>
<p>Posts about discontinued support for old versions of TLS <a href="https://central.sonatype.org/articles/2018/May/04/discontinue-support-for-tlsv11-and-below/">on Maven Central</a> and in the <a href="https://jfrog.com/knowledge-base/why-am-i-failing-to-work-with-jfrog-saas-service-with-tls-1-0-1-1/">Bintray knowledge base</a> explain the background for the necessity of these changes.</p>
<p>You may also find Gradle-specific details from <a href="https://github.com/gradle/gradle/issues/5740">gradle/gradle#5740</a> on GitHub.</p>
Using Gradle build cache with Kotlin2018-02-06T00:00:00-05:00https://blog.gradle.org/kotlin-build-cache-useEric Wendelin
<p>A <a href="https://blog.gradle.org/introducing-gradle-build-cache">build cache</a> allows Gradle to reuse task output from <em>any previous invocation, including those from other machines</em>. <a href="https://blog.jetbrains.com/kotlin/2018/01/kotlin-1-2-20-is-out/">Kotlin 1.2.21</a> allows Kotlin projects to make use of build caching.</p>
<p>The build cache works by storing compiled classes, test outputs, and other build artifacts in a cache, taking into account all task inputs, including input file contents, relevant classpaths, and task configuration.</p>
<p><img src="https://blog.gradle.org/images/kotlin-build-cache/build-cache-topological-diagram.png" alt="Build Cache topological diagram" /></p>
<p>This frequently results in <em>faster builds</em>. The following chart shows aggregated build time with and without the build cache for part of Gradle’s CI:</p>
<p><img src="https://blog.gradle.org/images/kotlin-build-cache/build-cache-total-build-minutes.png" alt="Build minutes saved with Gradle build cache" /></p>
<p>In this post, we’ll explain how you can use Gradle’s build cache to avoid unnecessary Kotlin compilation to speed up your builds.</p>
<h2 id="quick-demo-with-spek">Quick demo with Spek</h2>
<p>You can try out build caching with Gradle right now. Just follow these steps:</p>
<h4 id="clone-spek">Clone <a href="https://github.com/spekframework/spek">Spek</a></h4>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/spekframework/spek.git
<span class="nb">cd </span>spek
</code></pre></div></div>
<p>The Spek <code class="language-plaintext highlighter-rouge">2.x</code> branch (which is the default) already has all <a href="#enabling-the-build-cache-for-your-projects">prerequisites for build caching</a> that we’ll describe later.</p>
<h4 id="build-and-populate-cache">Build and populate cache</h4>
<p>The following command builds Spek and populates the local build cache.</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>❯ ./gradlew assemble --build-cache
BUILD SUCCESSFUL in 10s
21 actionable tasks: 21 executed
</code></pre></div></div>
<p>Using the <code class="language-plaintext highlighter-rouge">--build-cache</code> flag is one way to tell Gradle to store outputs in a separate task output cache.</p>
<h4 id="removechange-build-outputs">Remove/change build outputs</h4>
<p>This simulates being on another machine or perhaps making a change and <code class="language-plaintext highlighter-rouge">stash</code>ing it. The quickest way to demonstrate is use the <code class="language-plaintext highlighter-rouge">clean</code> task.</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>❯ ./gradlew clean
</code></pre></div></div>
<h4 id="re-build-and-resolve-from-build-cache">Re-build and resolve from build cache</h4>
<p>This time when we re-build, all Kotlin compiled sources are pulled from the build cache.</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>❯ ./gradlew assemble --build-cache
BUILD SUCCESSFUL in 2s
21 actionable tasks: 11 executed, 10 from cache
</code></pre></div></div>
<p>Voilà! You just used Gradle’s build cache to reuse Kotlin compiled classes instead of recompiling again! The build was about <em>5x faster</em>!</p>
<p>You can see from <a href="https://scans.gradle.com/s/mrdja32oqqpro/timeline?typeFilter=org.jetbrains.kotlin.gradle.tasks.KotlinCompile">this build scan</a> that Kotlin compile tasks were pulled from the build cache; <code class="language-plaintext highlighter-rouge">:jar</code> and <code class="language-plaintext highlighter-rouge">:processResources</code> tasks were not because it’s faster to generate JARs and copy files locally than pull from a cache. Note that caching <code class="language-plaintext highlighter-rouge">:test</code> tasks is also supported.</p>
<p>The Gradle build cache is particularly effective when a CI instance populates a <a href="https://docs.gradle.org/current/userguide/build_cache.html#sec:build_cache_configure_remote">shared build cache</a>, which developers can pull from. <a href="#further-reading">Links to more resources</a> for achieving this are listed below.</p>
<h2 id="enabling-the-build-cache-for-your-projects">Enabling the build cache for your projects</h2>
<p>I hope you’re excited to try this out on your project — you can follow these steps to enable the build cache.</p>
<p>First, you need to ensure you’re using Gradle 4.3 or above, so the Kotlin Gradle Plugin can opt-in to using new APIs in Gradle. You can upgrade Gradle easily using the <a href="https://docs.gradle.org/current/userguide/gradle_wrapper.html">Gradle wrapper</a>.</p>
<p>Next, we need to ensure we are compiling with Kotlin version 1.2.20 or above. You might have something like this declared in your <code class="language-plaintext highlighter-rouge">buildscript {}</code> block in <code class="language-plaintext highlighter-rouge">build.gradle</code>:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dependencies</span> <span class="o">{</span>
<span class="n">classpath</span> <span class="s2">"org.jetbrains.kotlin:kotlin-gradle-plugin:1.2.21"</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Next, we need to tell Gradle to use the build cache. There are 3 ways to do this:</p>
<ul>
<li>Enable for the current build using <code class="language-plaintext highlighter-rouge">--build-cache</code> on the command-line.</li>
<li>Enable for the project by adding <code class="language-plaintext highlighter-rouge">org.gradle.caching=true</code> to <code class="language-plaintext highlighter-rouge">$PROJECT_ROOT/gradle.properties</code>.</li>
<li>Enable for all builds for the current user by adding <code class="language-plaintext highlighter-rouge">org.gradle.caching=true</code> to <code class="language-plaintext highlighter-rouge">$GRADLE_HOME/gradle.properties</code>.</li>
</ul>
<p><strong>NOTE:</strong> Android developers still need to do this even if <code class="language-plaintext highlighter-rouge">android.enableBuildCache=true</code> is set, because Gradle’s build cache is separate from the Android build cache.</p>
<p>We can optionally take advantage of the build cache from IDEs by delegating run and test actions to Gradle.</p>
<h2 id="enabling-build-caching-in-intellij">Enabling build caching in IntelliJ</h2>
<p>If you use IntelliJ to execute Gradle actions, you will need to “Delegate IDE build/run actions to Gradle” in your IDE settings to take advantage of build caching when building and running tests from IntelliJ.</p>
<p><img src="https://blog.gradle.org/images/kotlin-build-cache/intellij-delegate-to-gradle.png" alt="Delegate IDE build/run to Gradle" /></p>
<p><strong>NOTE:</strong> Android Studio does this by default.</p>
<h2 id="caching-kapt-tasks">Caching kapt tasks</h2>
<p>Caching for <code class="language-plaintext highlighter-rouge">kapt</code> is currently disabled by default, even with <code class="language-plaintext highlighter-rouge">--build-cache</code>, because Gradle does not yet have a way to map inputs and outputs for annotation processors. You can explicitly enable use of the build cache for Kotlin annotation processing tasks by setting <code class="language-plaintext highlighter-rouge">useBuildCache</code> to <code class="language-plaintext highlighter-rouge">true</code> in <code class="language-plaintext highlighter-rouge">kapt</code> configuration.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">kapt</span> <span class="o">{</span>
<span class="n">useBuildCache</span> <span class="o">=</span> <span class="kc">true</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="further-reading">Further reading</h2>
<p>You can learn more about leveraging the Gradle build cache through these resources:</p>
<ul>
<li><a href="https://docs.gradle.org/current/userguide/build_cache.html#sec:build_cache_configure">Configuring the build cache</a></li>
<li><a href="https://docs.gradle.org/current/userguide/build_cache.html#sec:build_cache_configure_remote">Setting up a shared, remote build cache</a> ⚡️</li>
<li><a href="https://guides.gradle.org/using-build-cache/#debugging_and_diagnosing_cache_misses">Debugging build cache misses</a></li>
<li><a href="https://docs.gradle.org/current/userguide/build_cache.html#sec:task_output_caching_details">Developing cacheable custom tasks</a></li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>Compiling Kotlin code using <code class="language-plaintext highlighter-rouge">kotlin-gradle-plugin</code> version 1.2.20 and above can take advantage of Gradle’s <code class="language-plaintext highlighter-rouge">--build-cache</code> feature to speed up your development cycle. Work continues to expand the set of tasks that support build caching.</p>
<p><em>Onward!</em></p>
Introducing the new C++ plugins2018-01-10T00:00:00-05:00https://blog.gradle.org/introducing-the-new-cpp-pluginsAdam Murdoch
<p>This post introduces some new plugins for C++ that we’ve been working on. These plugins can build C++ libraries and applications. They work on macOS, Linux, and Windows with GCC, Clang and Visual C++/Visual Studio.</p>
<p>The plugins will eventually replace the software model plugins and take advantage of many new features baked into Gradle core, such as a rich dependency management engine, <a href="https://blog.gradle.org/introducing-gradle-build-cache">build cache</a>, <a href="https://blog.gradle.org/introducing-composite-builds">composite builds</a>, <a href="https://guides.gradle.org/using-the-worker-api/">finer grained parallel execution</a>, <a href="https://gradle.com/build-scans">build scans</a>, and more. For background, see our post on the <a href="https://blog.gradle.org/state-and-future-of-the-gradle-software-model">State and Future of the Gradle Software Model</a>.</p>
<p>We welcome any feedback you may have about these plugins. You can leave feedback on the <a href="https://discuss.gradle.org/">Gradle forums</a> or raise issues on the <a href="https://github.com/gradle/gradle-native">Gradle native</a> GitHub repository.</p>
<h2 id="building-an-application">Building an application</h2>
<p>You can find all of the samples from this post in the <a href="https://github.com/gradle/native-samples">Gradle native samples</a> GitHub repository. Let’s look at building a <a href="https://github.com/gradle/native-samples/tree/master/cpp/application">simple application</a>.</p>
<!--break-->
<p>The build script should look familiar to anyone who has used Gradle’s Java plugins:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s1">'cpp-application'</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This application has no dependencies, and the C++ source files and headers live in the default location: the <code class="language-plaintext highlighter-rouge">src/main/cpp</code> directory. Since this is Gradle, you can easily configure the source locations to match whatever layout your project has, including the common pattern of putting everything in one directory.</p>
<p>Here’s the result of running <code class="language-plaintext highlighter-rouge">./gradlew assemble</code> on this sample:</p>
<p><img src="/images/introducing-the-new-cpp-plugins/01-application.gif" alt="./gradlew assemble" /></p>
<p>Take a look at the build scan for this build to see what happened in more detail.</p>
<p><a href="https://scans.gradle.com/s/2fwbsmwmrvlqi/timeline"><img src="/images/introducing-the-new-cpp-plugins/compile-debug-cpp-timeline.png" alt="compile debug timeline" /></a></p>
<p>The plugins automatically find the compiler, linker and other tools to build the application. The result ends up installed in the <code class="language-plaintext highlighter-rouge">build/install</code> directory ready to run.</p>
<h2 id="ide-support">IDE support</h2>
<p>Xcode is currently supported for C++ projects. You can just run <code class="language-plaintext highlighter-rouge">./gradlew xcode</code> and open the generated workspace. Support for generating Visual Studio solutions will be added early this year and support for other IDEs will be gradually added after that.</p>
<p>Here is the result of running <code class="language-plaintext highlighter-rouge">./gradlew xcode</code> on the sample:</p>
<p><img src="/images/introducing-the-new-cpp-plugins/02-xcode.gif" alt="./gradlew xcode" /></p>
<p>This is how the workspace looks in Xcode:</p>
<p><img src="/images/introducing-the-new-cpp-plugins/02-xcode-ide.png" alt="Xcode integration" /></p>
<h2 id="dependencies">Dependencies</h2>
<p>The plugin uses Gradle’s <a href="https://docs.gradle.org/current/userguide/artifact_dependencies_tutorial.html">dependency management</a> features, just like other plugins such as the Java or Android plugins. This means, for example, transitive dependencies work just fine.</p>
<p>Let’s add a dependency on a library to the application. In <a href="https://github.com/gradle/native-samples/tree/master/cpp/binary-dependencies">this sample</a>, the C++ library is downloaded from a Maven repository. You don’t have to install the library anywhere manually, and everyone who runs the build will use the version specified in the build script, rather than whatever version happens to be installed on their machine.</p>
<p>The build script defines a Maven repository and declares a dependency on another sample C++ library:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">repositories</span> <span class="o">{</span>
<span class="n">maven</span> <span class="o">{</span>
<span class="c1">// In this sample, we used a local Maven repository,</span>
<span class="c1">// but Maven Central or Artifactory server can be used.</span>
<span class="n">url</span> <span class="s1">'http://localhost:8000/'</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">dependencies</span> <span class="o">{</span>
<span class="n">implementation</span> <span class="s1">'org.gradle.cpp-samples:math:1.5'</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here is the result of running <code class="language-plaintext highlighter-rouge">./gradlew assemble</code>. Gradle downloads the headers and shared library binary and compiles and links against these:</p>
<p><img src="/images/introducing-the-new-cpp-plugins/03-dependency-management.gif" alt="./gradlew assemble" /></p>
<p>The build scan shows more detail, including the downloads.</p>
<p><a href="https://scans.gradle.com/s/lglez2sq5zcik/performance/networkActivity"><img src="/images/introducing-the-new-cpp-plugins/cpp-assemble-network.png" alt="app assemble build scan network activity" /></a></p>
<p>Here is how this project looks in Xcode:</p>
<p><img src="/images/introducing-the-new-cpp-plugins/03-xcode-ide.png" alt="Xcode integration" /></p>
<h2 id="tests">Tests</h2>
<p>Basic unit testing is supported out of the box. Here is a <a href="https://github.com/gradle/native-samples/tree/master/cpp/simple-library">sample that uses Google Test</a>, downloaded from a Maven repository. We published the binaries using this <a href="https://github.com/gradle/googletest">fork</a> of Google Test, which simply adds a Gradle build.</p>
<p>The build script declares a dependency on Google test and a Maven repository that can be used to locate the Google test binaries:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s1">'cpp-unit-test'</span>
<span class="o">}</span>
<span class="n">repositories</span> <span class="o">{</span>
<span class="n">maven</span> <span class="o">{</span>
<span class="n">url</span> <span class="s1">'https://repo.gradle.org/gradle/libs-snapshots-local/'</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">dependencies</span> <span class="o">{</span>
<span class="c1">// Currently we have to encode the operating system and architecture in</span>
<span class="c1">// the dependency name. This will disappear in later releases</span>
<span class="n">testImplementation</span> <span class="s1">'org.gradle.cpp-samples:googletest_macosx_x86-64_4.5:1.9.0-SNAPSHOT'</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here is the result of running <code class="language-plaintext highlighter-rouge">./gradlew check</code>. Gradle downloads the Google test library, compiles the C++ source and tests and then runs the tests:</p>
<p><img src="/images/introducing-the-new-cpp-plugins/04-google-test.gif" alt="./gradlew check" /></p>
<p><a href="https://gradle.com/s/ox73aii7ieebi"><img src="/images/introducing-the-new-cpp-plugins/cpp-google-test.png" alt="build scan for math check" /></a></p>
<p>Richer reporting, build scan support, parallel execution and filtering for Google Test will be added this year with support for other C++ testing frameworks after that.</p>
<h2 id="fast-builds">Fast builds</h2>
<p>The plugins can produce debug and release builds of the application or library using Gradle’s new variant-aware dependency management, so that debug builds are compiled and linked against debug library binaries, and release builds are compiled and linked against release library binaries. When you build the debug build, which is the default, Gradle builds <em>only</em> the debug builds of the libraries that you need, rather than building everything.</p>
<p>Developer and CI builds are fast. C++ compilation is a cacheable task, so you can avoid unnecessary and long compilation times when using the <a href="https://docs.gradle.org/current/userguide/build_cache.html">build cache</a>. <a href="https://gradle.com/build-cache">Gradle Enterprise</a> comes with a build cache backend. You don’t need to use the <code class="language-plaintext highlighter-rouge">--parallel</code> option as Gradle does incremental and parallel compilation and linking by default.</p>
<p>Let’s run some clean builds that use the build cache:</p>
<p><img src="/images/introducing-the-new-cpp-plugins/05-build-cache.gif" alt="./gradlew assemble with build cache" /></p>
<p>You can see that the second build is faster, as the result is fetched from the build cache rather than recompiled. Build scans for <a href="https://gradle.com/s/grhgj6t3t2bpq">non-cached build</a> and <a href="https://gradle.com/s/rgbum6d7eysu6">cached build</a>.</p>
<p><img src="/images/introducing-the-new-cpp-plugins/cached-vs-non-cached.png" alt="cached vs non-cached assemble" /></p>
<h2 id="publishing-c-libraries">Publishing C++ libraries</h2>
<p>The new plugins can publish C++ libraries to a Maven or Ivy repository. Support for other kinds of repositories will be added later. Here is the <a href="https://github.com/gradle/native-samples/tree/master/cpp/simple-library">build</a> for the library we saw earlier.</p>
<p>The build script adds a Maven repository to publish the binaries to:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">plugins</span> <span class="o">{</span>
<span class="n">id</span> <span class="s1">'cpp-library'</span>
<span class="n">id</span> <span class="s1">'maven-publish'</span>
<span class="o">}</span>
<span class="n">group</span> <span class="o">=</span> <span class="s1">'org.gradle.cpp-samples'</span>
<span class="n">version</span> <span class="o">=</span> <span class="s1">'1.5'</span>
<span class="n">publishing</span> <span class="o">{</span>
<span class="n">repositories</span> <span class="o">{</span>
<span class="n">maven</span> <span class="o">{</span>
<span class="c1">// In this sample, we used a local maven repository,</span>
<span class="c1">// but Maven Central or Artifactory server can be used.</span>
<span class="n">url</span> <span class="s1">'http://localhost:8000/'</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here is the result of running <code class="language-plaintext highlighter-rouge">./gradlew publish</code>:</p>
<p><img src="/images/introducing-the-new-cpp-plugins/06-publishing.gif" alt="./gradlew publish" /></p>
<h2 id="composite-builds">Composite builds</h2>
<p>Composite builds also work the same as in Java projects. This <a href="https://github.com/gradle/native-samples/tree/master/cpp/composite-build">sample using a composite build</a> combines builds so they can be worked on together:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">rootProject</span><span class="o">.</span><span class="na">name</span> <span class="o">=</span> <span class="s1">'app'</span>
<span class="n">includeBuild</span> <span class="s1">'list-library'</span>
<span class="n">includeBuild</span> <span class="s1">'utilities-library'</span>
</code></pre></div></div>
<p>Here’s the result in Xcode. The application and the libraries it uses are available to edit, build and test together:</p>
<p><img src="/images/introducing-the-new-cpp-plugins/07-xcode-ide.png" alt="Xcode integration" /></p>
<p>Finally, it’s easy to set up a CI build for our application. We’ve added <a href="https://github.com/gradle/native-samples/blob/master/.travis.yml">configuration for Travis CI</a>.</p>
<p><a href="https://travis-ci.org/gradle/native-samples"><img src="/images/introducing-the-new-cpp-plugins/native-samples-travis-ci.png" alt="Native Samples Travis CI" /></a></p>
<h2 id="your-feedback-wanted">Your Feedback Wanted</h2>
<p>These plugins are a work in progress and have some limitations. For example, binary publishing doesn’t understand operating system or architecture yet. We’ll continue to improve these plugins, make them stable, and eventually will deprecate the software model plugins.</p>
<p>Please try these plugins out and let us know what you think. The easiest way to get started is to clone the <a href="https://github.com/gradle/native-samples">native samples</a> repository and follow the instructions. Our samples use a <a href="https://gradle.org/nightly/">Gradle nightly build</a>, so you’ll see the latest and greatest developments there. Some changes are already showing up in the <a href="https://docs.gradle.org/4.5-rc-1/release-notes.html#c/c++-compilation-improvements">4.5 release candidates</a>.</p>
<p>We’d love to hear what you think works well, what’s confusing, and what is missing that would block you from using Gradle to build C++ software. You can also leave feedback on the <a href="https://discuss.gradle.org/">Gradle forums</a> or raise issues on the <a href="https://github.com/gradle/gradle-native">Gradle native</a> GitHub repository.</p>
Multi-release JARs - Good or bad idea?2017-12-19T00:00:00-05:00https://blog.gradle.org/mrjarsCédric Champeau
<p>With Java 9 came a new feature of the Java runtime called multi-release jars. For us at Gradle, it’s probably one of the most controversial additions to the platform. TL/DR, we think it’s a wrong answer to a real problem. This post will explain why we think so, but also explain how you can build such jars if you really want to.</p>
<p><a href="http://openjdk.java.net/jeps/238">Multi-release JARs</a>, aka MRJARs, are a new feature of the Java platform, included in the Java 9 JDK. In this post, we will elaborate on the significant risks of adopting this technology and provide how one can produce and consume multi-release JARs with Gradle, if desired.</p>
<p>In a nutshell, multi-release jars allow you to package several versions of the same class, for consumption by different runtimes. For example, if you run on JDK 8, the Java runtime would use the Java 8 version of the class, but if you run on Java 9, it would use the Java 9 specific implementation. Similarly, if a version is built for the upcoming Java 10 release, then the runtime would use it instead of the Java 9 and default (Java 8) versions.</p>
<!--break-->
<h2 id="use-cases-for-multi-release-jars">Use Cases for multi-release JARs</h2>
<ul>
<li>
<p>Optimized runtime. This answers a problem that lots of developers have faced in real world: when you develop an application, you don’t know in what runtime it’s going to be executed. However, you know that for some runtimes you can implement optimized versions of the same class. For example, imagine that you want to display the Java version number that your application is currently executed on. For Java 9, you can use the <code class="language-plaintext highlighter-rouge">Runtime.getVersion</code> method. However, this is a new method only available if you run on Java 9+. If you target more runtimes, say, Java 8, then you need to parse the <code class="language-plaintext highlighter-rouge">java.version</code> property. So you end up with 2 different implementations of the same feature.</p>
</li>
<li>
<p>Conflicting APIs : Another common use case is to handle conflicting APIs. For example, you need to support 2 different runtimes, but one has deprecated APIs. There are currently 2 widely used solutions to this problem:</p>
<ul>
<li>The first one is to use reflection. One could for example define a <code class="language-plaintext highlighter-rouge">VersionProvider</code> interface, then 2 concrete classes <code class="language-plaintext highlighter-rouge">Java8VersionProvider</code> and <code class="language-plaintext highlighter-rouge">Java9VersionProvider</code>, the right one being loaded at runtime (note that funnily to be able to choose between the 2 you might have to parse the version number!). A variant of this solution is just to have a single class, but different methods, accessing and calling different methods by reflection.</li>
<li>A more advanced solution would be to use method handles for this, if it is technically applicable. Most likely, you would see reflection as both painful to implement and slow, and you would most likely be right.</li>
</ul>
</li>
</ul>
<h2 id="well-known-alternatives-to-multi-release-jars">Well known alternatives to multi-release JARs</h2>
<p>The second solution, easier to maintain and reason about, is to provide 2 different jars, aimed at 2 different runtimes. Basically, you would write the 2 implementations for the same class in your IDE, and it’s the build tool responsibility to compile, test and package them correctly into 2 different artifacts. This is the approach that some tools like Guava or Spock for example have been using for years. But it’s also what some languages like Scala need. Because there are so many variants of the compiler and the runtime, that binary compatibility is almost impossible to maintain.</p>
<p>But there are more reasons to prefer separate jars:</p>
<ul>
<li>a jar is <em>just</em> packaging
<ul>
<li>it’s an artifact of the build that happens to package classes, but not only: resources would typically be bundled into a jar too. Packaging, as well as processing resources, has a cost. What we’re trying to do with Gradle is to improve the performance of builds, and reduce the amount of time a developer has to wait to see results of compilation, tests, and in general the whole build process. By forcing to build a jar too early in the process, you create a redundant point of synchronization. For example, to compile downstream consumers, the only thing the consumer needs is the .class files. It doesn’t need the jar, nor does it need the resources in the jar. Similarly, to execute tests, all Gradle needs is the class files, plus the resources. There’s no need to actually <em>create</em> the jar to execute tests. The jar is only needed once an external consumer will require it (in short, publishing). But as soon as you consider the artifact as a requirement, then you’re blocking some tasks from running concurrently, and you’re slowing down the whole build. While for small projects this might not be an issue, for enterprise scale builds, this is a major blocker.</li>
</ul>
</li>
<li>more importantly, as an artifact, a jar shouldn’t carry information about dependencies.
<ul>
<li>There’s absolutely no reason why the runtime dependencies of your Java 9 specific class would be the same as the Java 8 one. In our very simplistic example they would, but for larger project this is wrong modeling: typically, users would import a backport library of a Java 9 feature, and use it to implement the Java 8 version of the class. However, if you package both versions in the same jar, then you’re mixing things that don’t have the same dependency trees into a single artifact. It means, typically, that if you happen to run on Java 9, you’re bringing a dependency that you would never ever use. Worse, it can (and will) pollute your classpath, possibly creating conflicts for consumers.</li>
</ul>
</li>
</ul>
<p>Eventually, for a single project, you can produce different jars, aimed at different usages:</p>
<ul>
<li>one for the API</li>
<li>one for Java 8 runtime</li>
<li>one for Java 9</li>
<li>one with native bindings</li>
<li>…</li>
</ul>
<p>Abuse of the <code class="language-plaintext highlighter-rouge">classifier</code> leads to inconsistent things being referred to using the same mechanism. Typically, the <code class="language-plaintext highlighter-rouge">sources</code> or <code class="language-plaintext highlighter-rouge">javadocs</code> jars are posted as classifiers, but don’t really have any dependency.</p>
<ul>
<li>we don’t want to create a mismatch depending on <em>how</em> you get your classes. In other words, using multi-release jars have the side effect that consuming from a jar and consuming from a class directory are no longer equivalent. There’s a semantic difference between the 2, which is terrible!</li>
<li>depending on the tool that is going to create the jar, you may produce inconsistent jars! The only tool so far that guarantees that if you package the same class twice in a jar, both of them have the same public API, is the <code class="language-plaintext highlighter-rouge">jar</code> tool itself. Which, for lots of good reasons is not necessarily used by build tools, or even users. A jar, in practice, is just an envelope. It’s a zip in disguise. So depending on how you build it, you would have different behavior, or you could just produce wrong artifacts and never notice.</li>
</ul>
<h2 id="better-ways-to-manage-separate-jars">Better ways to manage separate JARs</h2>
<p>The main reason developers don’t use separate jars is that they are impractical to produce and consume. The fault is on build tools, which, until Gradle, have dramatically failed at handling this. In particular, developers who have used this solution had no other choice than relying on the very poor <code class="language-plaintext highlighter-rouge">classifier</code> feature of Maven to publish additional artifacts. However, classifiers are very bad at modelling the complexity of the situation. They are used for a variety of different aspects, from publishing sources, documentation, javadocs, to publishing <em>variants</em> of a library (<code class="language-plaintext highlighter-rouge">guava-jdk5</code>, <code class="language-plaintext highlighter-rouge">guava-jdk7</code>, …) or different usages (api, fat jar, …). And in practice, there’s no way to indicate that the dependency tree of a <code class="language-plaintext highlighter-rouge">classifier</code> is not the one of the project itself. In other words, the POM is broken, as it represents both how the component is built, and what artifacts it produces. Say that you want to produce 2 different jars: one classic jar, and one <code class="language-plaintext highlighter-rouge">fat jar</code> that bundles all dependencies. In practice Maven would consider that the 2 artifacts have equal dependency trees, even if it’s plain wrong! It’s super obvious in this case, but the situation is exactly the same with multi-release jars!</p>
<p>The solution is to handle variants properly. That’s what we call variant-aware dependency management, and Gradle knows how to do it. So far this feature has only been enabled for Android development, but we’re currently developing it for Java and native too!</p>
<p>Variant-aware dependency management is the idea that modules and artifacts are different beasts. With the same source files, you can target different runtimes, with different requirements. For the native world it has been obvious for years: we compile for i386 and amd64, and there’s no way you can mix the dependencies of a i386 library with the ones of arm64! Transposed to the Java world, it means that if you target Java 8, you should produce a java 8 version of your jar, with classes targeting the Java 8 class format. This artifact would have metadata attached so that Java 8 consumers know what dependencies to use. And if you target Java 9, then the Java 9 dependencies would be selected. It’s as simple as that (well, in practice it’s not because the runtime is only one dimension of the variants, and you can combine multiple).</p>
<p>Of course, nobody has ever done this before because it’s complex to handle: Maven would for sure never let you do such complex thing. But Gradle makes it possible. And the good news is that we’re also developing a new metadata format that will let consumers know which variant they should use. Simply said, the build tool needs to deal with the complexity of compiling, testing, packaging, but also consuming such modules. For example, say that you want to support Java 8 and Java 9 as runtimes. Then, <em>ideally</em>, you need to compile 2 versions of your library. Which means 2 different compilers (to avoid using the Java 9 APIs while targeting Java 8), 2 different class directories, and 2 different jars in the end. But also, you will probably want to test the 2 different runtimes. Or, you might want to build the 2 jars, but still want to test what the behavior of the Java 8 version is when executed on a Java 9 runtime (because, it may happen in production!).</p>
<p>We’ve made <a href="https://github.com/gradle/gradle/blob/master/subprojects/docs/src/docs/design/gradle-module-metadata-specification.md">significant progress</a> towards modelling this, and even if we’re not ready yet, it explains why we are not so keen on using multi-release jars: while they fix a problem, they are fixing it the wrong way, and Maven Central is going to be bloated with libraries that do not declare their dependencies properly!</p>
<h2 id="how-to-create-a-multi-release-jar-with-gradle">How to create a multi-release JAR with Gradle</h2>
<p>It’s not ready so what should I do? The good news is that the path to generate correct artifacts is the same. Until this new feature is ready for the Java ecosystem, you have 2 different options:</p>
<ul>
<li>do it the old way, using reflection or distinct jars.</li>
<li>use multi-release jars, (being aware that you may take the wrong decision here, even if there are good use cases)</li>
</ul>
<p>Whatever solution you choose, separate jars route or the multi-release jars, both use the same setup. Multi-release jars are only the wrong (default) packaging: they should be an <em>option</em>, not a goal. Technically, the source layout is the same for both separate jars and external jars. This <a href="https://github.com/melix/mrjar-gradle">repository</a> explains how you can create a multi-release jar with Gradle, but here is how it works in a nutshell.</p>
<p>First, you must understand that we as developers often have a very bad habit: we tend to run Gradle (or Maven) using the same Java version that the artifacts you want to produce. Sometimes it’s even worse, when we use a more recent version to run Gradle, and compile using an older API level. But there’s no good reason to do this. Gradle supports <a href="https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_cross_compilation">cross-compilation</a>. It allows you to explain where a JDK is found, and fork compilation to use this specific JDK to compile a component. A reasonable way to setup different JDKs is to configure the path to the JDKs through environment variables, which is what we are doing <a href="https://github.com/melix/mrjar-gradle/blob/master/jdks.gradle">in this file</a>. Then we only need to configure Gradle to use the appropriate JDK <a href="https://github.com/melix/mrjar-gradle/blob/master/jdks.gradle#L12-L23">based on the source/target compatibility</a>. It’s worth noting that starting from JDK 9, it’s no longer necessary to provide older JDKs to perform cross-compilation. A new option, <code class="language-plaintext highlighter-rouge">-release</code>, does exactly that. Gradle will <a href="https://github.com/melix/mrjar-gradle/blob/master/jdks.gradle#L35">recognize this option and configure the compiler accordingly</a>.</p>
<p>The second key concept is the notion of <a href="https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_source_sets">source set</a>. A source set represents a set of sources that are going to be compiled together. A jar is built from the result of the compilation of one or more source sets. For each source set, Gradle will automatically create a corresponding compile task that you can configure. This means that if we have sources for Java 8 and sources for Java 9, then they should live in separate source sets. That’s what we do by creating a <a href="https://github.com/melix/mrjar-gradle/blob/master/build.gradle#L4-L10">Java 9 specific source set</a> that will contain the specialized version of our class. This matches reality, and doesn’t force you to create a separate project like Maven would require. But more importantly, it allows us to precisely configure <em>how</em> this source set is going to compile.</p>
<p>Part of the challenge of multiple versions of a single class is that it’s very rare that such a class is totally independent from the rest of the code (it has dependencies onto classes that are found in the main source set). For example, its API would use classes that don’t need to have Java 9 specific sources. Yet, you don’t want to recompile all those common classes, nor do you want to package Java 9 versions of all those classes. They are really shared and should stay separate. This is what <a href="https://github.com/melix/mrjar-gradle/blob/master/build.gradle#L19">this line</a> is about: it will configure the dependency between the Java 9 source set and the main source set, making sure that when we compile the Java 9 specific version, all common classes are on compile classpath.</p>
<p>The <a href="https://github.com/melix/mrjar-gradle/blob/master/build.gradle#L22-L30">next step</a> is really simple: we need to explain to Gradle that the main source set is going to target Java 8 language level, and that the Java 9 source set is going to target Java 9 language level.</p>
<p>All the steps we have described so far allow you both approaches described previously: publishing separate jars, or publishing a multi-release jar. Since this is the topic of this blog post, let’s see how we can now tell Gradle that we will only generate a multi-release jar:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">jar</span> <span class="o">{</span>
<span class="n">into</span><span class="o">(</span><span class="s1">'META-INF/versions/9'</span><span class="o">)</span> <span class="o">{</span>
<span class="n">from</span> <span class="n">sourceSets</span><span class="o">.</span><span class="na">java9</span><span class="o">.</span><span class="na">output</span>
<span class="o">}</span>
<span class="n">manifest</span><span class="o">.</span><span class="na">attributes</span><span class="o">(</span>
<span class="s1">'Multi-Release'</span><span class="o">:</span> <span class="s1">'true'</span>
<span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This configuration block does 2 separate things: bundle the Java 9 specific classes into the <code class="language-plaintext highlighter-rouge">META-INF/versions/9 directory</code>, which is expected in a MRJar add the multi-release flag to the manifest.</p>
<p>And that’s it, you’ve built your first MRJar! However we’re not done yet, unfortunately. If you are familiar with Gradle, you would know that if you apply the <code class="language-plaintext highlighter-rouge">application</code> plugin you can also run the application directly with a <code class="language-plaintext highlighter-rouge">run</code> task. However, because as usual Gradle tries to perform the minimal amount of work to do what you need, the <code class="language-plaintext highlighter-rouge">run</code> task is wired to use the class directories as well as the processed resources directories. And for multi-release jars, that’s a problem, because you need the jar now! So instead of relying on this plugin we have no choice but <a href="https://github.com/melix/mrjar-gradle/blob/master/build.gradle#L46-L50">creating our own task</a>, which is another reason why not use multi-release jars.</p>
<p>Last but not least, we said we probably also want to test the 2 versions of our class. For this, you have no choice but using forked VMs, because there’s no equivalent to the <code class="language-plaintext highlighter-rouge">-release</code> flag for the Java runtime. The idea, here, is that you write a single unit test, but it’s going to be executed twice: once with Java 8, the other with Java 9 runtime. This is the only way to make sure that your substituted classes work properly. By default, Gradle only creates a single test task, and it will also use the class directories instead of the jar. So we need to do two things:
create a Java 9 specific test task
configure both test tasks so that they use the jar and specific Java runtimes</p>
<p>This can be achieved simply by doing this:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">test</span> <span class="o">{</span>
<span class="n">dependsOn</span> <span class="n">jar</span>
<span class="kt">def</span> <span class="n">jdkHome</span> <span class="o">=</span> <span class="n">System</span><span class="o">.</span><span class="na">getenv</span><span class="o">(</span><span class="s2">"JAVA_8"</span><span class="o">)</span>
<span class="n">classpath</span> <span class="o">=</span> <span class="n">files</span><span class="o">(</span><span class="n">jar</span><span class="o">.</span><span class="na">archivePath</span><span class="o">,</span> <span class="n">classpath</span><span class="o">)</span> <span class="o">-</span> <span class="n">sourceSets</span><span class="o">.</span><span class="na">main</span><span class="o">.</span><span class="na">output</span>
<span class="n">executable</span> <span class="o">=</span> <span class="n">file</span><span class="o">(</span><span class="s2">"$jdkHome/bin/java"</span><span class="o">)</span>
<span class="n">doFirst</span> <span class="o">{</span>
<span class="n">println</span> <span class="s2">"$name runs test using JDK 8"</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">task</span> <span class="nf">testJava9</span><span class="o">(</span><span class="nl">type:</span> <span class="n">Test</span><span class="o">)</span> <span class="o">{</span>
<span class="n">dependsOn</span> <span class="n">jar</span>
<span class="kt">def</span> <span class="n">jdkHome</span> <span class="o">=</span> <span class="n">System</span><span class="o">.</span><span class="na">getenv</span><span class="o">(</span><span class="s2">"JAVA_9"</span><span class="o">)</span>
<span class="n">classpath</span> <span class="o">=</span> <span class="n">files</span><span class="o">(</span><span class="n">jar</span><span class="o">.</span><span class="na">archivePath</span><span class="o">,</span> <span class="n">classpath</span><span class="o">)</span> <span class="o">-</span> <span class="n">sourceSets</span><span class="o">.</span><span class="na">main</span><span class="o">.</span><span class="na">output</span>
<span class="n">executable</span> <span class="o">=</span> <span class="n">file</span><span class="o">(</span><span class="s2">"$jdkHome/bin/java"</span><span class="o">)</span>
<span class="n">doFirst</span> <span class="o">{</span>
<span class="n">println</span> <span class="n">classpath</span><span class="o">.</span><span class="na">asPath</span>
<span class="n">println</span> <span class="s2">"$name runs test using JDK 9"</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">check</span><span class="o">.</span><span class="na">dependsOn</span><span class="o">(</span><span class="n">testJava9</span><span class="o">)</span>
</code></pre></div></div>
<p>Now if you run the <code class="language-plaintext highlighter-rouge">check</code> task, Gradle will compile each source set using the proper JDK, build a multi-release jar, then run unit tests using this jar on both JDKs. Future versions of Gradle will help you do this in a more declarative way.</p>
<h2 id="conclusion">Conclusion</h2>
<p>In conclusion, we’ve seen that multi-release jars address a real problem that a significant number of library designers face. However, we think this is the wrong solution to the problem. Correct modeling of dependencies, as well as coupling of artifacts and variants, and not forgetting performance (ability to execute more tasks concurrently) make them a poor man’s solution to a problem we are fixing the right way, using variant-aware dependency management. However, we reckon that for simple use cases, knowing that variant-aware dependency management for Java is not yet completed, it may be convenient to produce such a jar. In that case, and only in that case, this post helped you understand how you can do this, and how the philosophy of Gradle differs from Maven in this case (source set vs project).</p>
<p>Finally, we don’t deny that there are cases where multi-release jars do make sense: applications for which the runtime is not known in advance, for example, but those are exceptional and should be considered as such. Most issues are for <em>library designers</em>: we’ve covered common problems they face, and how multi-release JARs attempt to solve some of them. Modeling dependencies correctly as variants improves performance (via finer-grained parallelism) and reduces maintenance overhead (avoiding accidental complexity) over the use of multi-release JARs. Your situation may dictate that MRJARs be used; rest assured that it’s still supported by Gradle. See <a href="https://github.com/melix/mrjar-gradle">this mrjar-gradle example project</a> to try this today.</p>
State of Gradle Java 9 Support2017-10-17T00:00:00-04:00https://blog.gradle.org/java-9-support-updateEric Wendelin
<p>This post provides an overview of Gradle’s Java 9 support, touching on runtime, cross-compilation, MRJARs, and Jigsaw modules support. We’ve fielded lots of questions since Java 9 was released last month, and decided it best to answer here.</p>
<h2 id="what-gradle-supports-as-of-version-421">What Gradle supports as of version 4.2.1</h2>
<p>As of Gradle 4.2.1, building and running Java applications using major distributions of JDK 9 such as Oracle JDK9, OpenJDK9 and Azul JDK9 is fully supported. Further, cross-compilation (built by JDK9 but runs on JDK8) is supported.</p>
<p>Some builds will break when upgrading to Java 9, regardless of build tool used. The Java team have made good and necessary changes to the JDK to facilitate better software architecture and security, but this has meant removing access to some APIs. Even if your project is ready, some tools and Gradle plugins have not yet been updated to work with Java 9.</p>
<p>There is no convenience methods for consuming and assembling Multi-Release JARs, but you can take a look at this <a href="https://github.com/melix/mrjar-gradle">MRJAR-gradle example</a> if you desire to use them.</p>
<!--break-->
<h2 id="java-modules-aka-jigsaw-support">Java Modules aka Jigsaw Support</h2>
<p>If you’re not yet familiar with the Java 9 Platform Module System, also known as Project Jigsaw, you should read <a href="http://openjdk.java.net/projects/jigsaw/quick-start">Project Jigsaw: Module System Quick-Start Guide</a>. The motivation and terminology are well explained in <a href="http://openjdk.java.net/projects/jigsaw/spec/sotms/">The State of the Module System</a>.</p>
<p>A module is defined as “a named, self-describing collection of code and data” whereby packages are treated as code boundaries, and are explicitly <em>exported</em> and <em>required</em>. Non-exported packages are not visible to module consumers, and further 2 modules cannot export the same packages, nor can they have the same internal packages. This means that packages cannot be “split” or duplicated between multiple modules, or compilation will fail.</p>
<p>Here is a <a href="https://guides.gradle.org/building-java-9-modules/">guide that shows how to use Java modules</a> with Gradle today. It walks you through the steps necessary to tell Gradle to use the <code class="language-plaintext highlighter-rouge">modulepath</code> and not <code class="language-plaintext highlighter-rouge">classpath</code> when compiling Java sources and patch modules for testing purposes.</p>
<p>A bottom-up approach (convert libraries with no dependencies first) is recommended if you wish to incrementally convert to Java 9 modules. After all, modules are consumable as regular JARs. Be mindful of <a href="http://blog.joda.org/2017/05/java-se-9-jpms-automatic-modules.html">automatic modules</a> when “legacy” JARs are added to the <code class="language-plaintext highlighter-rouge">modulepath</code>.</p>
<h2 id="achieving-encapsulation-with-the-java-library-plugin">Achieving encapsulation with the Java Library Plugin</h2>
<p>One of the 2 major goals of the Java 9 module system is to provide better software architecture through strong encapsulation. Gradle 3.4 introduced the <a href="https://docs.gradle.org/current/userguide/java_library_plugin.html">Java Library Plugin</a> that enforces strong encapsulation for libraries by separating <code class="language-plaintext highlighter-rouge">api</code> dependencies (those meant to be exposed to consumers) from <code class="language-plaintext highlighter-rouge">implementation</code> dependencies whose internals are not leaked to consumers.</p>
<p>This, of course, does not eliminate use of Java classpaths as is <a href="http://openjdk.java.net/projects/jigsaw/spec/sotms/">stated as another goal of Java modules</a>. You can learn about the motivation and usage of the Java Library Plugin in <a href="https://blog.gradle.org/incremental-compiler-avoidance">this post</a>. It’s worth noting that the Java Library Plugin is useful for projects using Java 7 and above — you do not need to migrate to Java 9 to have some stronger encapsulation.</p>
<p>Here’s what this means in practice, given this example library:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">apply</span> <span class="nl">plugin:</span> <span class="s1">'java-library'</span>
<span class="n">name</span> <span class="o">=</span> <span class="s1">'mylibrary'</span>
<span class="n">group</span> <span class="o">=</span> <span class="s1">'com.mycompany'</span>
<span class="n">dependencies</span> <span class="o">{</span>
<span class="n">api</span> <span class="nf">project</span><span class="o">(</span><span class="s1">':model'</span><span class="o">)</span>
<span class="n">implementation</span> <span class="s1">'com.google.guava:guava:18.0'</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Let’s presume we have an application that uses <code class="language-plaintext highlighter-rouge">mylibrary</code>.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyApplication</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// This does not compile using 'java-library' plugin</span>
<span class="nc">Set</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">strings</span> <span class="o">=</span> <span class="n">com</span><span class="o">.</span><span class="na">google</span><span class="o">.</span><span class="na">common</span><span class="o">.</span><span class="na">collect</span><span class="o">.</span><span class="na">ImmutableSet</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"Hello"</span><span class="o">,</span> <span class="s">"Goodbye"</span><span class="o">);</span>
<span class="c1">// This compiles and runs</span>
<span class="nc">Foo</span> <span class="n">foo</span> <span class="o">=</span> <span class="n">com</span><span class="o">.</span><span class="na">mycompany</span><span class="o">.</span><span class="na">model</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">Foo</span><span class="o">();</span>
<span class="c1">// This also compiles and runs</span>
<span class="nc">Class</span> <span class="n">clazz</span> <span class="o">=</span> <span class="nc">MyApplication</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getClassLoader</span><span class="o">().</span><span class="na">loadClass</span><span class="o">(</span><span class="s">"com.mycompany.model.internal.Foo"</span><span class="o">);</span>
<span class="nc">Foo</span> <span class="n">foo</span> <span class="o">=</span> <span class="o">(</span><span class="nc">Foo</span><span class="o">)</span> <span class="n">clazz</span><span class="o">.</span><span class="na">getConstructor</span><span class="o">().</span><span class="na">newInstance</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>You can see that you get some of the benefits by adopting the Gradle’s Java Library plugin. If you are migrate to Java modules, you can use this rough mapping:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">implementation</code> dependency => <code class="language-plaintext highlighter-rouge">requires</code> module declaration</li>
<li><code class="language-plaintext highlighter-rouge">api</code> dependency => <code class="language-plaintext highlighter-rouge">requires transitive</code> module declaration</li>
<li><code class="language-plaintext highlighter-rouge">runtimeOnly</code> dependency => <code class="language-plaintext highlighter-rouge">requires static</code> module declaration</li>
</ul>
<h2 id="next-steps">Next Steps</h2>
<p><a href="https://blog.gradle.org/subscribe">Stay tuned</a> for updates on first-class Java modules support in Gradle.</p>
<p>You can use the <a href="https://guides.gradle.org/building-java-9-modules/">Building Java 9 Modules guide</a> to learn how to use Java modules with Gradle today.</p>
Join Gradle at KotlinConf 20172017-10-10T00:00:00-04:00https://blog.gradle.org/kotlinconf-2017Eric Wendelin
<p>Kotlin is the #1 rising Gradle “related topic” <a href="https://trends.google.com/trends/explore?geo=US&q=%2Fm%2F080c0g9">according to Google Trends</a>. We want to improve user experience developing Kotlin applications with Gradle, and that requires great guides — but we need your help! Tell us which unwritten Kotlin Gradle guide you think would be most useful, and you’ll have <em>a chance to get a free ticket</em> to <a href="https://kotlinconf.com/">KotlinConf</a>! See instructions below.</p>
<p style="text-align: center">
<img src="/images/kotlinconf-wide.png" alt="KotlinConf 2-3 Nov 2017, San Francisco, Pier 27" />
</p>
<p>We are proud to be a silver partner for <a href="https://kotlinconf.com/">KotlinConf</a>, which is just 3 weeks away! Hans Dockter will be presenting <a href="https://www.kotlinconf.com/speakers/#speaker=hans-dockter">Building Kotlin Applications at Scale</a> and some of the <a href="https://github.com/gradle/kotlin-dsl">Gradle Kotlin DSL</a> team will be on-site to talk Kotlin with you.</p>
<!--break-->
<h3 id="about-the-kotlinconf-2017-ticket-raffle">About the KotlinConf 2017 ticket raffle</h3>
<p>We are giving away 2 standard tickets to the conference. You’ll be responsible for getting to the conference, so please don’t accept a ticket if you cannot attend.</p>
<p>All you need to do is file an issue on the <a href="https://github.com/gradle/guides/">gradle/guides GitHub repo</a> with a unique and useful suggestion for a Gradle guide around building Kotlin with Gradle. For inspiration, you can check out the <a href="https://github.com/gradle/guides/issues/95">proposal for the Building Kotlin JVM Libraries guide</a>, which was published recently.</p>
<p><del>We’d be happy to get lots of good ideas from you, but we’ll enter your name once. Winners will be chosen at random on <em>Monday, October 16th</em>. We’ll contact you if you’ve been selected, and you’ll have 3 days to confirm you can attend.</del></p>
<p><strong>UPDATE</strong>: Thank you everyone! The contest is now closed.</p>
<p>If you’re interested in contributing to the Kotlin/Gradle community, just adding 👍 s on ideas you think would be good is helpful, but even more so submitting a PR with a sample or typo fix is most welcome. Remember the goal: make Kotlin development awesome for everyone!</p>
<p>Good luck! We hope to see you at <a href="https://kotlinconf.com/">KotlinConf</a>!</p>
State and future of the Gradle Software Model2017-08-24T00:00:00-04:00https://blog.gradle.org/state-and-future-of-the-gradle-software-modelDaniel Lacasse
<p>We’ve received many inquiries about the status and direction of <a href="https://docs.gradle.org/current/userguide/pt06.html">Gradle’s Software Model</a>, especially from users building native libraries and applications.</p>
<p>In this blog post, we will explain the current state and future of the Software Model, and in particular how it relates to native development with Gradle. A lot of exciting improvements are planned for the remainder of 2017; see the <a href="#a-way-forward">roadmap below</a>.</p>
<h2 id="situation-with-the-software-model">Situation with the Software Model</h2>
<p>In a nutshell, the Software Model is a very declarative way to describe how a piece of software is built and the other components it needs as dependencies in the process. It also provides a new, rule-based engine for configuring a Gradle build. When we started to implement the Software Model we set ourselves the following goals:</p>
<!--break-->
<ul>
<li>Improve configuration and execution time performance.</li>
<li>Make customizations of builds with complex tool chains easier.</li>
<li>Provide a richer, more standardized way to model different software ecosystems</li>
</ul>
<p>As we were developing the Software Model, the Gradle engineering team constantly tried to dogfood the concepts into existing software ecosystems. The Gradle plugins for building native applications is currently fully-based on the Software Model. Similarly, experimental Software Model-based plugins were developed for ecosystems like Android and Java.</p>
<p>Gradle adoption in the native ecosystem is picking up, and so is our investment. Since it’s inception, <a href="https://docs.gradle.org/current/userguide/native_software.html">Gradle’s native support</a> has proved itself as a welcome alternative for builds using Make. With its declarative and expressive model, support for different tool chains and platforms as well as features like parallel compilations, it offers a revolutionary way for building native libraries and applications.</p>
<p>It took us longer than expected to evolve the new configuration and Software Model and make it as powerful as the current Gradle model for Java and Android. Meanwhile, Gradle adoption skyrocketed, there are many complex builds out there using the current model and a vibrant ecosystem of 1500+ community plugins as well. We underestimated the complexity for those builds and plugins to migrate to the new model and saw understandable resistance with many of our partners to undergo this migration.</p>
<p>In hindsight, the scope of the new software and configuration model was too big. That is why at the Gradle Summit 2016, Hans Dockter <a href="https://www.youtube.com/watch?v=lFugHBBsGe4&feature=youtu.be&t=2098">announced</a> that we were backporting many of its features to the current model. One year later, most of the features for the Java and Android ecosystem have been backported. This includes variant-aware dependency resolution and <a href="https://docs.gradle.org/current/userguide/java_library_plugin.html#sec:java_library_separation">separation of API and implementation</a> for Java components. Those features were game changers in terms of <a href="https://blog.gradle.org/incremental-compiler-avoidance">work avoidance</a> and <a href="https://blog.gradle.org/blazing-fast-android-builds">performance</a>. Furthermore, we found other ways to drastically improve Gradle configuration performance, with more to come. There is no longer any need for a drastic, incompatible change in how Gradle builds are configured.</p>
<h2 id="a-way-forward">A way forward</h2>
<p>You may therefore be wondering what is happening to the Software Model. We’re in the process of porting the configuration DSL of the native support to the current model. So the declarative nature and strong modelling language will be the same. <em>The rule engine that was part of the Software Model will be deprecated.</em> Everything under the model block will be ported as extensions to the current model. Native users will no longer have a separate extension model compared to the rest of the Gradle community, and they will be able to make use of the new variant aware dependency management.</p>
<p>What does the roadmap look like? Here are the areas of focus until the end of the year:</p>
<ul>
<li><strong>Current model support for native</strong>. New sets of plugins based on the current model are in development and are improved with every nightly release. They still need more work to achieve feature parity and stability, but already provide a lot of functionality. <a href="https://github.com/gradle/native-samples">Try them out</a> and give us feedback.</li>
<li><strong>Parallel-by-default for compile and link tasks</strong>. Performance improvements are planned for the native ecosystem by enabling parallelism-by-default to compile and link tasks. This will have a positive impact on everyone building native with Gradle.</li>
<li><strong>Transitive dependency resolution</strong>. We are porting this powerful feature from our JVM ecosystem to help native developers declare rich dependencies between native projects.</li>
<li><strong>New native plugins on current model</strong>. Our plan is to have plugins that have most of the functionality of the Software Model plugins and will also have substantial new features like build caching and external source dependencies for native.</li>
<li><strong>Improved tool chain support</strong>. We are looking at ironing out some of the wrinkles with tool chain declaration which is particularly important for embedded development.</li>
</ul>
<p>For the most complete and up-to-date progress, we recommend having a look at the <a href="https://github.com/gradle/gradle-native">gradle-native project</a>, the home for the native ecosystem feature planning.</p>
<p>User migration from Software Model plugins to the new ones will be pretty seamless. All core native tasks will be reused and the tool chain concept will be ported to the current model. We expect that a lot of your build logic can be simply reused. We will support the Software Model-based plugins for an extended period of time to ensure everyone has a successful migration.</p>
<p>If you are currently using, or are planning to use, Gradle to build native projects, by all means keep doing so. Gradle’s native support has proven time and time again to be more performant, flexible, and easier to use than currently available tools.</p>
<h3 id="exciting-things-afoot">Exciting things afoot</h3>
<p>Today, we’re working on IDE integration and XCTest support, with out-of-the-box HTML report generation and <a href="https://scans.gradle.com/">full build scan support</a>.
Tool chain definition will also be improved to allow easier integration with alternative tool chain families; this is especially exciting for users invested in the embedded world.</p>
<p>For multi-repository developers, you will be happy to learn that <a href="https://docs.gradle.org/current/userguide/composite_builds.html">composite builds</a> will work for all native projects.</p>
<p><a href="https://scans.gradle.com/s/2o3l5zm63frhk"><img src="/images/cpp-build-scan.png" alt="C++ Build Scan" /></a></p>
<p>The new plugins will integrate with the <a href="https://github.com/gradle/kotlin-dsl">Kotlin DSL</a> which gives Gradle users proper IDE support including auto-completion and refactoring.</p>
<p>We will first implement the complete workflow for native development in the current model without any customization - i.e. no platforms, build types or tool chains configuration. By workflow we mean everything related to building binaries, as well as testing, packaging, deploying and integration with your favorite IDE. At first, the workflow will work for the most common cases. In the subsequent releases, we will proceed by adding customization incrementally to the whole workflow.</p>
<h2 id="community-involvement">Community involvement</h2>
<p>Our native community is one of the most active and engaged on <a href="https://discuss.gradle.org/c/help-discuss">the forum</a>, and we want to encourage and grow that engagement even more.
Please keep helping each other find the answers you are seeking in the forum, but also engage with us by, trying the <a href="https://github.com/gradle/native-samples">various native sample projects</a>, subscribing to the <a href="https://github.com/gradle/gradle-native/issues">gradle-native project</a> and filing issues, voting on issues that are most important to you, and even consider submitting pull requests if you’re excited to roll up your sleeves and pitch in.</p>
<p>The best ways to stay passively up-to-date on native platform support are to subscribe to the <a href="https://newsletter.gradle.org/">monthly newsletter</a> or more frequent announcements <a href="https://twitter.com/gradle">on Twitter</a>.</p>
<p>We look forward to working with you to develop the best native build tool!</p>
Blazing Fast Android Builds2017-05-17T00:00:00-04:00https://blog.gradle.org/blazing-fast-android-buildsStefan Oehme
<p>At Google I/O today, the Android Studio team released the first preview version of the <a href="https://android-developers.googleblog.com/2017/05/android-studio-3-0-canary1.html">Android Gradle plugin 3.0</a>, based on <a href="https://docs.gradle.org/4.0-milestone-2/release-notes.html">Gradle 4.0 M2</a>. It brings major performance improvements, especially for builds with plenty of subprojects. In this blog post, we will explain what you can expect from this preview version and how the Android Studio and Gradle team achieved these improvements. Before diving into this let’s look back at what goals led to the creation of the current Android build system.</p>
<h2 id="the-complexity-of-mobile-development">The Complexity of Mobile Development</h2>
<p>Developing mobile applications is inherently more complex than building traditional web or server applications of similar size. An app needs to support a wide array of devices with different peripherals, different screen sizes, and comparatively slow hardware. The popular freemium model adds another layer of variety, requiring different code paths for free and paid versions of the app. In order to provide a fast, slim app for every device and target audience, the build system needs to do a lot of the heavy lifting up front.</p>
<p>To improve developer productivity and to reduce runtime overhead, the Android build tools provide several languages and source generators, e.g. Java, RenderScript, AIDL and Native code. Packaging an app together with its libraries involves highly customizable merging and shrinking steps. The Android Studio team was faced with the challenge of automating all of these without exposing the underlying complexity to developers. Developers can focus on writing their production code.</p>
<!--break-->
<p>Last but not least, developers expect a build tool to manage their dependencies, be extensible and provide deep IDE integration.</p>
<p>Gradle is ideally suited for those challenges and the Android Studio team created a fantastic Android build tool on top of the Gradle platform.</p>
<h2 id="the-performance-challenge">The performance challenge</h2>
<p>No matter how elegant and extensible the plugin and no matter how seamless the IDE integration, when things take too long, developers become unproductive and frustrated. The Android Studio team has made steady progress on performance over the last years. The emulators became much faster, the time to deploy an app decreased by orders of magnitude with Instant Run and other improvements. These steps have now exposed the build itself as the final bottleneck. The Android Studio team and the Gradle team have continuously improved the performance of the plugin and the platform, but so far this has not been enough. Fundamental design issues preventing great performance.</p>
<p>So Gradle Inc. and Google teamed up in late 2016 to get this situation under control. The work was split up into three areas:</p>
<ul>
<li>General improvements to Gradle and its Java support: Faster up-to-date checking, compile avoidance, stable incremental compilation and parallel dependency downloads.</li>
<li>General improvements to the Android tools, like dex and code shrinking, including incremental dexing.</li>
<li>New APIs for variant aware dependency management in Gradle and an Android plugin that uses these new APIs.</li>
</ul>
<p>The latter allowed the Android Studio team to finally get rid of a lot of inefficient workarounds that they had to build because of these missing APIs.</p>
<p>To understand why variant aware dependency management is so important, imagine you have an app which depends on a single library. Both of them support ARM and x86 architectures, both have a free and a paid version and both of them can be built for debug and production. This creates a total of 8 variants. But at any given point in time, a developer is only working on exactly one variant, e.g. the “free x86 debug” variant.</p>
<p>Up until now, the Android plugin had to inspect the app’s dependencies very early in the build lifecycle to select the right variant of the library to build. This early phase is called configuration time, during which Gradle determines what tasks it needs to run in what order. More work at configuration time means slower builds no matter which tasks the user selected. It also affects how long it takes to synchronize the build with the IDE. The Android plugin’s eager dependency inspection lead to a combinatorial explosion of configuration time as more subprojects were added to a build.</p>
<p>This completely changes with Gradle’s new variant aware dependency management. The Android plugin can now provide matching strategies for the different variant dimensions (like product flavor and build type), which Gradle uses during dependency resolution to select the correct variant of the upstream library. This completely removes the need to resolve dependencies at configuration time and also allows the Android plugin to only build the parts of the library that the app needs.</p>
<p>In a particularly <a href="https://github.com/gradle/perf-android-large/">large app</a> with 130 subprojects, the time it took to configure the project dropped from 3 minutes to 10 seconds with Android 2.3 tools to under 2 seconds with Android 3.0. The clean build time dropped from over 5 minutes to about 1 minute. The effect on incremental builds is dramatic when combined with new <a href="https://blog.gradle.org/incremental-compiler-avoidance">compile avoidance</a> functionality. Making a single-line change and assembling the project is down to about 9 seconds. For monolithic projects these numbers won’t be as impressive, but they show that the build system now works very efficiently with modularized apps.</p>
<p><img src="/images/android-performance.png" alt="Android performance comparison" /></p>
<p>Last but not least, the Android Studio team is going to make the Android plugin 3.0 compatible with the <a href="https://blog.gradle.org/introducing-gradle-build-cache">Gradle build cache</a>. The build cache allows build outputs to be reused across clean builds and across machine boundaries. This means that developers can reuse build outputs generated by CI and build pipelines can reuse results from earlier stages. It also speeds up switching between feature branches on developer machines. Preliminary tests are promising, the clean build for the large Android app mentioned above dropped from 60s to about 20s when using the cache.</p>
<h2 id="give-it-a-try">Give it a try</h2>
<p>The Android Studio team has written up a comprehensive <a href="https://developer.android.com/studio/preview/features/new-android-plugin-migration.html">migration guide</a>. There may be compatibility issues with community plugins, as many of them depended on internals that work differently now.</p>
<p>If you are developing Android projects, give the preview a try and tell us how much your build times improved out of the box. Try modularizing your app a bit more and splitting <code class="language-plaintext highlighter-rouge">api</code> and <code class="language-plaintext highlighter-rouge">implementation</code> dependencies for even bigger performance gains. You can use <a href="http://scans.gradle.com/">Build Scans</a> and its <a href="https://scans.gradle.com/s/n2hf3xwzucwny/timeline">timeline</a> view to get deep insight into the performance of your build, which tasks were executed and how long they took.</p>
<p>If you are an Android plugin author, the new version might require some changes for your plugin to stay compatible. Please <a href="https://issuetracker.google.com/issues/new?component=192709&template=842921">file an issue</a> if you encounter any problems while migrating.</p>
<h2 id="whats-next">What’s next?</h2>
<p>You can expect more improvements on the Gradle side. For instance, we are currently working on parallel task execution by default.</p>
<p>It is also safe to expect more performance smartness from the Android Studio team including Android Studio optimizations to do as little work as possible when syncing the project. The Gradle and Android Studio teams are collaborating on this as well.</p>
<p>Support for community plugins will improve as the alpha versions mature and plugin authors adjust to it. The more people provide feedback, the faster these great improvements can be released as stable.</p>
Introducing Gradle Build Cache Beta2017-04-10T00:00:00-04:00https://blog.gradle.org/introducing-gradle-build-cacheSterling Greene
<p>Introduced in <a href="https://docs.gradle.org/3.5/release-notes.html">Gradle 3.5</a> to reduce build time.</p>
<h2 id="what-does-it-do">What does it do?</h2>
<p>The build cache reuses the outputs of Gradle tasks locally and shares task outputs between machines. In many cases, this will accelerate the average build time.</p>
<p>The build cache is complementary to Gradle’s incremental build features, which optimizes build performance for local changes that have not been built already. Many Gradle tasks are designed to be <a href="https://blog.gradle.org/introducing-incremental-build-support">incremental</a>, so that if the inputs and outputs of the task do not change, Gradle can skip the task. Even when the task’s inputs have changed, <a href="https://blog.gradle.org/incremental-compiler-avoidance">some tasks</a> can rebuild only the parts that have changed. Of course, these techniques only work if there are already outputs from previous local builds. In the past, building on fresh checkouts or executing “clean” builds required building everything from scratch again, even if the result of those builds had already been created locally or on another machine (such as the continuous integration server).</p>
<p>Now, Gradle uses the inputs of a task as a key to uniquely identify the outputs for a task. With the build cache feature enabled, if Gradle can find that key in a build cache, Gradle will skip task execution and directly copy the outputs from the cache into the build directory. This can be much faster than executing the task again.</p>
<!--break-->
<p>In particular, if you’re using a continuous integration server, you can configure Gradle to push task outputs to a shared build cache. When a developer builds, task outputs already built on CI are copied to the developer’s machine. This can greatly improve the developer’s local build experience.</p>
<p>When using the local build cache, instead of rebuilding large parts of the project whenever you switch branches, Gradle can skip task execution and pull the previous outputs from the local cache.</p>
<h2 id="how-does-it-work">How does it work?</h2>
<p>A cacheable Gradle task is designed to declare everything that can affect the output of the task as an input. Gradle calculates a build cache key by hashing over all of the inputs to a task. That build cache key uniquely identifies the outputs of the task. This is an opt-in feature for each task implementation, so not every task is cacheable or needs to be. Several built-in Gradle tasks (<code class="language-plaintext highlighter-rouge">JavaCompile</code>, <code class="language-plaintext highlighter-rouge">Test</code>, <code class="language-plaintext highlighter-rouge">Checkstyle</code>) have caching enabled to speed up the typical Java project.</p>
<p>The build cache key for a task takes into account:</p>
<ul>
<li>The values of all inputs defined by the task via annotations (e.g. <code class="language-plaintext highlighter-rouge">@InputFiles</code>) or the runtime <a href="https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/TaskInputs.html">TaskInputs</a> API.</li>
<li>The contents (and relative paths) of any file inputs.</li>
<li>The classpath of the task, which includes any plugins and Gradle version used.</li>
<li>The classpath of any task actions, which can include the build script.</li>
</ul>
<p>When the build cache feature is enabled, Gradle will check if any of the configured build caches contain a match for the task’s build cache key when a task is not up-to-date.
If Gradle does not find a match, the task will be executed as normal. After execution, Gradle will gather all of the task’s outputs and push them to the build caches, if configured to do so.
If Gradle does find a match, the task’s outputs are deleted and the previous outputs are copied into the output directories.</p>
<h2 id="does-it-help">Does it help?</h2>
<p>We have been using the build cache for the Gradle CI builds since November 2016. We also have some partners who have been trying the build cache in their builds. We can’t share their data directly, but they’ve seen similar improvements in CI and developer builds as we have. On average, we see a 25% reduction in total time spent building each commit, but some commits are even better (80% reduction) and the median build saw a 15% reduction.</p>
<p><img src="/images/stage3-cached-vs-non-cached-pct.png" alt="Stage 3 %-Improved" /></p>
<p>Here’s another look at the number of minutes spent between the cached and non-cached builds for Gradle. You can see how the reductions translates into about 90 minutes saved in a 360 minute build for us.</p>
<p><img src="/images/stage3-cached-vs-non-cached.png" alt="Stage 3 comparison" /></p>
<p>The build cache is a generic feature that avoids re-executing a task when it can, so builds large and small can benefit in some way. The structure of your project will influence how much you can gain overall. If your project consists of a single monolithic module, Gradle has other features that may also help, such as <a href="https://blog.gradle.org/incremental-compiler-avoidance">incremental compilation</a> or <a href="https://blog.gradle.org/introducing-composite-builds">composite builds</a>. We’ll provide more information about how to get the most out of the build cache in a future blog post and at the <a href="https://summit.gradle.com/session/39174">Gradle Summit</a>.</p>
<h2 id="make-your-build-faster-today">Make your build faster today</h2>
<p>The <a href="https://docs.gradle.org/3.5/release-notes.html#faster-builds-with-the-gradle-build-cache">Gradle 3.5 release</a> is the first release to include the build cache feature.</p>
<p>We expect that the build cache feature will have general availability in the next release, but we would like for every project to give the build cache beta a try. To do that, we’d like you to try 3 things for us.</p>
<h3 id="1-try-it-on-a-simple-project">1) Try it on a simple project</h3>
<p>After upgrading to 3.5, pick a simple Java project and run:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gradle --build-cache clean assemble
gradle --build-cache clean assemble
</code></pre></div></div>
<p>The second build should be faster because some task outputs are reused from the first build. These outputs will be pulled from your local build cache, which is located in a directory in your GRADLE_USER_HOME.</p>
<h3 id="2-try-to-share-outputs-between-machines">2) Try to share outputs between machines</h3>
<p>To use a shared, remote cache, we provide a <a href="https://github.com/gradle/gradle/tree/REL_3.5/subprojects/docs/src/samples/buildCache/developer-ci-setup">recommended configuration</a> that uses your continuous integration builds to populate a shared build cache and allows all developers to pull from that build cache.</p>
<p>You’ll need a remote build cache backend to share between developers.
We provide a <a href="https://hub.docker.com/r/gradle/build-cache-node/">build cache node</a> docker image which operates as a remote Gradle build cache, and can connect with <a href="https://gradle.com/enterprise">Gradle Enterprise</a> for centralized management.
The cache node can also be used without a Gradle Enterprise installation with restricted functionality.</p>
<h3 id="3-give-us-feedback">3) Give us feedback</h3>
<p>If you have feedback, we’d love to hear it. If you have a <a href="https://gradle.com/scans/get-started">build scan</a> you can share, that’s even better.</p>
<p>We’re excited to get the Gradle Build Cache feature out for feedback in Gradle 3.5, but we know there’s more we need to do to make the build cache stable and performant. We have some <a href="https://docs.gradle.org/3.5/userguide/build_cache.html#sec:task_output_caching_known_issues">known issues</a> that you should check before <a href="https://github.com/gradle/gradle/issues/new?labels=in:build-cache">raising new issues on GitHub</a>.</p>
<p>At this time, we don’t recommend that you leave the build cache enabled for production builds without understanding the risks. There are <a href="https://docs.gradle.org/3.5/userguide/build_cache.html#sec:task_output_caching_known_issues">known issues</a> that can cause your builds to fail or produce incorrect output, but your feedback on the types of problems or successes are very valuable to maturing the build cache feature. You can configure the build cache in your build and enable it on a trial basis by setting <code class="language-plaintext highlighter-rouge">org.gradle.caching=true</code> or running with <code class="language-plaintext highlighter-rouge">--build-cache</code> without impacting all builds.</p>
<p>For dogfooding the build cache for Gradle, we used a separate CI job to run a build with the build cache enabled. This allowed us to compare the build times with and without the build cache for the same set of changes.</p>
<h3 id="thanks-and-roadmap">Thanks and roadmap</h3>
<p>After trying the build cache, you’ll probably have some questions about why more parts of your build are not cacheable. Regardless of the build cache backend you are using, <a href="https://gradle.com/enterprise/releases/2017.2">Gradle Enterprise 2017.2</a> comes with features to understand build cache usage and behavior by collecting data, whether the build cache is enabled or not. Build scans keep track of the reason that a task was not cached. A task might not be cached if it has particular problems, like if it has no outputs or cacheability is not enabled for it. You can search the build scan timeline for each of these reasons.</p>
<p>In future versions of Gradle and Gradle Enterprise, we’ll collect more information related to the build cache and task cacheability, to make it easier to diagnose build failures or poor build cache performance.</p>
<p>For the next release, Gradle 4.0, we intend to focus on making the build cache safe to enable for all well behaved builds and providing feedback for situations where Gradle cannot safely cache outputs from a task. This also means we’ll be providing a well-behaved local build cache and several validation checks.</p>
<p>For the releases following that, we intend to spend time on expanding our documentation and <a href="https://guides.gradle.org">Gradle guides</a> to make it easier for you to cache more tasks and develop cache-friendly tasks.</p>
<p>Thanks for your continued help and support. Please consider <a href="#make-your-build-faster-today">making your build faster</a> with the build cache with the three steps we outline above.</p>
Announcing Gradle Enterprise 2017.12017-03-08T00:00:00-05:00https://blog.gradle.org/announcing-gradle-enterprise-2017.1Craig Atkinson
<p>We are excited to announce the release of Gradle Enterprise 2017.1. This release includes many new features and bug fixes, further expanding the build insights that build scans provide you and your team. Here are some of the highlights of this release. <a href="https://gradle.com/contact-us">Contact us</a> if you’re interested in a demo or a trial.</p>
<h2 id="easily-find-changes-to-dependencies-between-two-builds">Easily find changes to dependencies between two builds</h2>
<p>Dependency changes between builds can be a common source of problems. For example, upgrading a version of one library can unintentionally bring in different versions of transitive dependencies into your project. In turn, these newer versions can cause you all kinds of frustration by breaking compatibility with other libraries that your project uses.</p>
<p>The new build comparison feature allows you to quickly find dependency changes between builds, including differences in transitive dependencies.</p>
<!--break-->
<p>You can easily select two builds to compare:</p>
<p><img src="/images/ge-2017-1/scan-list-comparison-selection.png" alt="Select builds for dependency comparison" /></p>
<p>And quickly see the dependency differences between the two builds:</p>
<p><img src="/images/ge-2017-1/dependency-comparison.png" alt="Dependency comparison" /></p>
<h2 id="visualize-your-builds-task-execution-with-the-timeline">Visualize your build’s task execution with the timeline</h2>
<p>When trying to make your build faster, it can be really helpful to know whether all processes are utilized efficiently. Are there optimization opportunities such as long-running tasks that could be split into smaller tasks and run in parallel? To find these optimization opportunities you first need to identify where the bottlenecks are in your build.</p>
<p>The new timeline feature gives you a visual representation of the tasks executed during your build. Using this visualization you can quickly identify bottleneck tasks in your build, places in your build where you could speed up execution by running more tasks in parallel, and other optimization opportunities.</p>
<p><img src="/images/ge-2017-1/timeline.jpg" alt="Timeline" /></p>
<p>You can also filter tasks by name/path, type and more, making it easy to inspect and highlight particular tasks.</p>
<p>Try out the timeline with <a href="https://scans.gradle.com/s/cqimzwzs3nflg/timeline" target="_blank">this example scan</a>.</p>
<h2 id="view-dependency-downloads">View dependency downloads</h2>
<p>Time spent downloading dependencies can have a significant impact on your build time. The new “Network Activity” tab in the “Performance” section shows all the downloads triggered by dependency resolution in your build, including the size of each download and how long it took.</p>
<p>You can identify big or slow downloads that are dragging down your build speed. Are there downloads from slow remote repositories that you could cache on-site? Or large downloads that are no longer needed in your build and could be removed entirely?</p>
<p>Also, you can see the overall number of downloads in your build, total download size, and average download speed across the downloads to quickly gauge overall network performance during your build.</p>
<p><img src="/images/ge-2017-1/network-activity.png" alt="Network activity" /></p>
<p>This feature requires the upcoming Gradle version 3.5 and build scan plugin 1.6 or later.</p>
<p>See network activity on <a href="https://scans.gradle.com/s/w7uodh22pjiam/performance/networkActivity" target="_blank">this example scan</a>.</p>
<h2 id="integrate-your-build-data-with-other-systems">Integrate your build data with other systems</h2>
<p>The new Export API provides a mechanism for consuming the raw build data that powers build scans. It is a HTTP interface based on Server Sent Events (SSE) that supports real time data integration. Libraries for consuming SSE streams are available for most programming languages.</p>
<p>The video below demonstrates a real time build duration dashboard built on the Export API. The code for this is available as part of the <a href="https://github.com/gradle/gradle-enterprise-export-api-samples" target="_blank">gradle-enterprise-export-api-samples</a> repository on GitHub.</p>
<iframe src="https://player.vimeo.com/video/202944447" width="660" height="372" frameborder="0" title="Gradle Enterprise - Export API" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen="">
</iframe>
<h2 id="see-why-a-task-wasnt-cacheable">See why a task wasn’t cacheable</h2>
<p>Gradle 3.3 introduced the build cache feature, which saves you time by reusing task outputs from other builds without needing to execute the task on your machine. For a given task to use the build cache, certain conditions must be met. Gradle Enterprise now indicates which tasks are cacheable and not cacheable.</p>
<p>To give you the opportunity to make more tasks cacheable and improve your build performance, you can see the reasons why tasks were not cacheable. The “Settings and Suggestions” tab of the “Performance” section now indicates if there were tasks that were not cacheable.</p>
<p><img src="/images/ge-2017-1/not-cacheable-tasks-suggestion.png" alt="Not-cacheable tasks suggestion" /></p>
<p>And in the new timeline view you can search for cacheable and non-cacheable tasks as well as see why individual tasks were not cacheable.</p>
<p><img src="/images/ge-2017-1/not-cacheable-task.png" alt="Not-cacheable task" /></p>
<p>This feature requires Gradle version 3.4 and build scan plugin 1.6 or later.</p>
<p>Try it out with <a href="https://scans.gradle.com/s/cqimzwzs3nflg/performance/suggestions" target="_blank">this example scan</a>.</p>
<h2 id="better-understand-task-performance">Better understand task performance</h2>
<p>Gradle can save you build time by not re-executing tasks that don’t need to be executed again. For example, tasks that are already up-to-date, or where the outputs can be pulled from the build cache.</p>
<p>The “Task Execution” tab of the “Performance” section summarizes which tasks were executed and which were avoided. The summary gives you an understanding how well cacheable your build currently is, making it easier for you to find optimization opportunities by tuning tasks to make them cacheable. You can also click from the summary into the timeline to see all tasks in a particular category.</p>
<p><img src="/images/ge-2017-1/task-execution-breakdown.png" alt="Task execution breakdown" /></p>
<p>Try it out with <a href="https://scans.gradle.com/s/ynmchyv645uiw/performance/execution" target="_blank">this example scan</a>.</p>
<h2 id="find-builds-by-absence-of-a-tag">Find builds by absence of a tag</h2>
<p>You can annotate a build scan with one or more tags to easily categorize the build. For example, to indicate which builds were executed on your continuous integration server.</p>
<p>Previously you could find scans that had one or more specific tags, and now you can also do the inverse - find scans that don’t have a specific tag. To do that, use the <code class="language-plaintext highlighter-rouge">not:</code> prefix when searching tags. For example, if you tag all of your continuous integration builds with the “CI” tag, you can find all non-CI builds by searching <code class="language-plaintext highlighter-rouge">not:CI</code>.</p>
<p><img src="/images/ge-2017-1/negative-tag-filtering.png" alt="Negative tag filtering" /></p>
<p>Please see this <a href="https://blog.gradle.org/custom-data-in-build-scans" target="_blank">Custom Data in Build Scans</a> post for more about how and when to use tags.</p>
<h2 id="find-builds-faster">Find builds faster</h2>
<p>Gradle Enterprise gives you the ability to find exactly the builds you need by filtering builds by project name, start time, outcome, and more. With the latest release, searching for build scans is now much faster - especially when you are searching through a large number of builds. This makes it faster to find exactly the builds you are looking for.</p>
<h2 id="try-it-today">Try it today!</h2>
<p>We hope you are as excited as we are about these great new features. <a href="https://gradle.com/contact-us">Contact us</a> today for a trial! You can also <a href="https://gradle.com/enterprise/releases/2017.1">check out the release notes</a> to see what else is new.</p>
Incremental Compilation, the Java Library Plugin, and other performance features in Gradle 3.42017-02-25T00:00:00-05:00https://blog.gradle.org/incremental-compiler-avoidanceCédric Champeau
<p>We are very proud to announce that the newly released Gradle 3.4 has significantly improved support for building Java applications, for all kind of users. This post explains in details what we fixed, improved and added. We will in particular focus on:</p>
<ul>
<li>Extremely fast incremental builds</li>
<li>The end of the dreaded compile classpath leakage</li>
</ul>
<p>The improvements we made can dramatically improve your build times. Here’s what we measured:</p>
<style>
#chart {
width: 100%
}
</style>
<div id="chart"></div>
<!-- Load c3.css -->
<link href="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.11/c3.min.css" rel="stylesheet" type="text/css" />
<!-- Load d3.js and c3.js -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.4/d3.min.js" charset="utf-8"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.11/c3.min.js"></script>
<script>
var chart = c3.generate({
data: {
rows: [
['Scenario','Maven 3.3.9','Gradle 3.3','Gradle 3.4'],
['Large project 1 change',20.80,13.10,1.30],
['Medium project 1 change',5.70,1.30,0.26],
['Multi project ABI-breaking change',26.80,15.80,3.30],
['Multi project ABI-compatible change',26.80,16.30,1.40]],
type: 'bar',
x: 'Scenario',
labels: true
},
axis: {
x: {
type: 'category'
},
y: {
label: 'seconds'
}
},
legend: {
position: 'right'
}
});
</script>
<p>The benchmarks are <a href="https://github.com/gradle/performance-comparisons">public</a>, and you can try them out yourself and are synthetic projects representing real world issues reported by our consumers. In particular, what matters in a continuous development process is being incremental (making a small change should never result in a long build):</p>
<!--break-->
<p>For those who work on a single project with lots of sources:</p>
<ul>
<li>changing a single file, in a big monolithic project and recompiling</li>
<li>changing a single file, in a medium-sized monolithic project and recompiling</li>
</ul>
<p>For multi-project builds:</p>
<ul>
<li>making a change in an ABI-compatible way (change the body of a method, for example, but not method signatures) in a subproject, and recompiling</li>
<li>making a change in an ABI-incompatible way (change a public method signature, for example) in a subproject, and recompiling</li>
</ul>
<p>For all those scenarios, Gradle 3.4 is <em>much</em> faster. Let’s see how we did this.</p>
<h2 id="compile-avoidance-for-all">Compile avoidance for all</h2>
<p>One of the greatest changes in Gradle 3.4 regarding Java support just comes for free: upgrade to Gradle 3.4 and benefit from <em>compile avoidance</em>. Compile avoidance is different from incremental compilation, which we will cover later. So what does it mean? It’s actually very simple. Imagine that your project <code class="language-plaintext highlighter-rouge">app</code> depends on project <code class="language-plaintext highlighter-rouge">core</code>, which itself depends on project <code class="language-plaintext highlighter-rouge">utils</code>:</p>
<p>In <code class="language-plaintext highlighter-rouge">app</code>:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Main</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">...</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">WordCount</span> <span class="n">wc</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WordCount</span><span class="o">();</span>
<span class="n">wc</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="nc">File</span><span class="o">(</span><span class="n">args</span><span class="o">[</span><span class="mi">0</span><span class="o">]);</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Word count: "</span> <span class="o">+</span> <span class="n">wc</span><span class="o">.</span><span class="na">wordCount</span><span class="o">());</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In <code class="language-plaintext highlighter-rouge">core</code>:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">WordCount</span> <span class="o">{</span> <span class="c1">// WordCount lives in project `core`</span>
<span class="c1">// ...</span>
<span class="kt">void</span> <span class="nf">collect</span><span class="o">(</span><span class="nc">File</span> <span class="n">source</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">IOUtils</span><span class="o">.</span><span class="na">eachLine</span><span class="o">(</span><span class="n">source</span><span class="o">,</span> <span class="nl">WordCount:</span><span class="o">:</span><span class="n">collectLine</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In <code class="language-plaintext highlighter-rouge">utils</code>:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">IOUtils</span> <span class="o">{</span> <span class="c1">// IOUtils lives in project `utils`</span>
<span class="kt">void</span> <span class="nf">eachLine</span><span class="o">(</span><span class="nc">File</span> <span class="n">file</span><span class="o">,</span> <span class="nc">Callable</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">action</span><span class="o">)</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">(</span><span class="nc">BufferedReader</span> <span class="n">reader</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">BufferedReader</span><span class="o">(</span><span class="k">new</span> <span class="nc">FileReader</span><span class="o">(</span><span class="n">file</span><span class="o">)))</span> <span class="o">{</span>
<span class="c1">// ...</span>
<span class="o">}</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">IOException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// ...</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Then, change the implementation of <code class="language-plaintext highlighter-rouge">IOUtils</code>. For example, change the body of <code class="language-plaintext highlighter-rouge">eachLine</code> to introduce the expected charset:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">IOUtils</span> <span class="o">{</span> <span class="c1">// IOUtils lives in project `utils`</span>
<span class="kt">void</span> <span class="nf">eachLine</span><span class="o">(</span><span class="nc">File</span> <span class="n">file</span><span class="o">,</span> <span class="nc">Callable</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">action</span><span class="o">)</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">(</span><span class="nc">BufferedReader</span> <span class="n">reader</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">BufferedReader</span><span class="o">(</span><span class="k">new</span> <span class="nc">InputStreamReader</span><span class="o">(</span><span class="k">new</span> <span class="nc">FileInputStream</span><span class="o">(</span><span class="n">file</span><span class="o">),</span> <span class="s">"utf-8"</span><span class="o">)</span> <span class="o">))</span> <span class="o">{</span>
<span class="c1">// ...</span>
<span class="o">}</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">IOException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// ...</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Now rebuild <code class="language-plaintext highlighter-rouge">app</code>. What happens? Until now, <code class="language-plaintext highlighter-rouge">utils</code> had to be recompiled, but then it also triggered the recompilation of <code class="language-plaintext highlighter-rouge">core</code> and eventually <code class="language-plaintext highlighter-rouge">app</code>, because of the dependency chain. It sounds reasonable at first glance, but is it really?</p>
<p>What changed in <code class="language-plaintext highlighter-rouge">IOUtils</code> is purely an internal detail. The implementation of <code class="language-plaintext highlighter-rouge">eachLine</code> changed, but its public API didn’t. Any class file previously compiled against <code class="language-plaintext highlighter-rouge">IOUtils</code> is still valid. Gradle is now smart enough to realize that. This means that if you make such a change, Gradle will only recompile <code class="language-plaintext highlighter-rouge">utils</code>, and nothing else! And while this example may sound simple, it’s actually a very common pattern: typically, a <code class="language-plaintext highlighter-rouge">core</code> project is shared by many subprojects, and each subproject has dependencies on different subprojects. A change to <code class="language-plaintext highlighter-rouge">core</code> would trigger a recompilation of all projects. With Gradle 3.4 this will no longer be the case, meaning that it recognizes ABI (<em>Application Binary Interface</em>) breaking changes, and will trigger recompilation only in that case.</p>
<p>This is what we call compilation avoidance. But even in the case when the compilation can not be avoided, Gradle 3.4 will make things much faster with the help of incremental compile.</p>
<h2 id="improved-incremental-compilation">Improved incremental compilation</h2>
<p>For years, Gradle has supported an experimental incremental compiler for Java. In Gradle 3.4, not only is this compiler stable, but we also have significantly improved both its robustness and performance! Use it now: we’re going to make it the default soon! To enable Java incremental compilation, all you need to do is to set it on the compile options:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tasks</span><span class="o">.</span><span class="na">withType</span><span class="o">(</span><span class="n">JavaCompile</span><span class="o">)</span> <span class="o">{</span>
<span class="n">options</span><span class="o">.</span><span class="na">incremental</span> <span class="o">=</span> <span class="kc">true</span> <span class="c1">// one flag, and things will get MUCH faster</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If we add the following class in project <code class="language-plaintext highlighter-rouge">core</code>:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">NGrams</span> <span class="o">{</span> <span class="c1">// NGrams lives in project `core`</span>
<span class="c1">// ...</span>
<span class="kt">void</span> <span class="nf">collect</span><span class="o">(</span><span class="nc">String</span> <span class="n">source</span><span class="o">,</span> <span class="kt">int</span> <span class="n">ngramLength</span><span class="o">)</span> <span class="o">{</span>
<span class="n">collectInternal</span><span class="o">(</span><span class="nc">StringUtils</span><span class="o">.</span><span class="na">sanitize</span><span class="o">(</span><span class="n">source</span><span class="o">),</span> <span class="n">ngramLength</span><span class="o">);</span>
<span class="o">}</span>
<span class="c1">// ...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>and this class in project <code class="language-plaintext highlighter-rouge">utils</code>:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">StringUtils</span> <span class="o">{</span>
<span class="kd">static</span> <span class="nc">String</span> <span class="nf">sanitize</span><span class="o">(</span><span class="nc">String</span> <span class="n">dirtyString</span><span class="o">)</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Imagine that we change the class <code class="language-plaintext highlighter-rouge">StringUtils</code> and recompile our project. You can easily see that we only need to recompile <code class="language-plaintext highlighter-rouge">StringUtils</code> and <code class="language-plaintext highlighter-rouge">NGrams</code> but not <code class="language-plaintext highlighter-rouge">WordCount</code>. <code class="language-plaintext highlighter-rouge">NGrams</code> is a dependent class of <code class="language-plaintext highlighter-rouge">StringUtils</code>. <code class="language-plaintext highlighter-rouge">WordCount</code> doesn’t use <code class="language-plaintext highlighter-rouge">StringUtils</code>, so why would it need to be recompiled? This is what the incremental compiler does: it analyzes the dependencies between classes, and only recompiles a class when it has changed, or one of the classes it depends on has changed.</p>
<p>Those of you who have already tried the incremental Java compiler before may have seen that it wasn’t very smart when a changed class contained a constant. For example, this class contains a constant:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SomeClass</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span> <span class="no">MAGIC_NUMBER</span> <span class="o">=</span> <span class="mi">123</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If this class was changed, then Gradle gave up and recompiled not just all the classes of that project but also all the classes in projects that depend on that project. If you wonder why, you have to understand that the Java compiler inlines constants like this. So when we analyze the result of compilation, and that the bytecode of a class contains the <em>literal</em> 123, we have no idea where the literal was defined. It could be in the class itself, or a constant of any dependency found anywhere on its classpath. In Gradle 3.4, we made that behavior much smarter, and only recompile classes which could <em>potentially</em> be affected by the change. In other words, if the class is changed, but the constant is not, we don’t need to recompile. Similarly, if the constant is changed, but that the dependents didn’t have a literal in their bytecode of the old value, we don’t need to recompile them: we would only recompile the classes that have <em>candidate literals</em>. This also means that not all constants are born equal: a constant value of <code class="language-plaintext highlighter-rouge">0</code> is much more likely to trigger a full recompilation when changed, than a constant value <code class="language-plaintext highlighter-rouge">188847774</code>…</p>
<p>Our incremental compiler is also now backed with in-memory caches that live in the Gradle daemon across builds, and thus make it significantly faster than it used to be: extracting the ABI of a Java class is an expensive operation that used to be cached, but on disk only.</p>
<p>If you combine all those incremental compilation improvements with the <em>compile avoidance</em> that we described earlier in this post, Gradle is now really fast when recompiling Java code. Even better, it also works for external dependencies. Imagine that you upgrade from <code class="language-plaintext highlighter-rouge">foo-1.0.0</code> to <code class="language-plaintext highlighter-rouge">foo-1.0.1</code>. If the only difference between the two versions of the library is, for example, a bugfix, and that the API hasn’t changed, compile avoidance will kick in and this change in an external dependency will not trigger a recompile of your code. If the new version of the external dependency has a modified public API, Gradle’s incremental compiler will analyze the dependencies of your project on <em>individual</em> classes of the external dependency, and only recompile where necessary.</p>
<h2 id="about-annotation-processors">About annotation processors</h2>
<p>Annotation processors are a very powerful mechanism that allows generation of code just by annotating sources. Typical use cases include dependency injection (<a href="https://google.github.io/dagger/">Dagger</a>) or boilerplate code reduction (<a href="https://projectlombok.org/">Lombok</a>, <a href="https://github.com/google/auto/tree/master/value">Autovalue</a>, <a href="https://github.com/JakeWharton/butterknife">Butterknife</a>, …). However, using annotation processors can have a very negative impact on the performance of your builds.</p>
<h3 id="what-does-an-annotation-processor-do">What does an annotation processor do?</h3>
<p>Basically, an annotation processor is a Java compiler <em>plugin</em>. It is triggered whenever the Java compiler recognizes an annotation that is handled by a processor. From the build tool point of view, it’s a black box: we don’t know what it’s going to do, in particular what files it’s going to generate, and where.</p>
<p>Therefore whenever the annotation processor implementation changes, Gradle needs to recompile everything. That is not that bad by itself, as this probably doesn’t happen very often. But for reasons explained soon things are much worse and Gradle has to disable compile avoidance when annotation processors are not declared explicitly. But first let’s understand what’s going on.
Typically today annotation processors are added to the compile classpath.</p>
<p>While Gradle can detect which jar contains annotation processors, what it cannot detect is which other jars in the compile classpath are used by the annotation processor implementation. They also have dependencies. That means potentially any change in the compile classpath may affect the behavior of the annotation processor in a way Gradle can not understand. Therefore any change in the compile classpath will trigger a full recompile and we are back to square one.</p>
<p>But there is a solution to this.</p>
<h4 id="explicitly-declaring-the-annotation-processor-classpath">Explicitly declaring the annotation processor classpath</h4>
<p>Should the fact that an annotation processor, which is a <em>compiler plugin</em> that uses external dependencies, influence your compile classpath? No, the dependencies of the annotation processor should <em>never</em> leak into your compile classpath. That’s why <code class="language-plaintext highlighter-rouge">javac</code> has a specific <code class="language-plaintext highlighter-rouge">-processorpath</code> option which is distinct from <code class="language-plaintext highlighter-rouge">-classpath</code>. Here is how you can declare this with Gradle:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">configurations</span> <span class="o">{</span>
<span class="n">apt</span>
<span class="o">}</span>
<span class="n">dependencies</span> <span class="o">{</span>
<span class="c1">// The dagger compiler and its transitive dependencies will only be found on annotation processing classpath</span>
<span class="n">apt</span> <span class="s1">'com.google.dagger:dagger-compiler:2.8'</span>
<span class="c1">// And we still need the Dagger annotations on the compile classpath itself</span>
<span class="n">compileOnly</span> <span class="s1">'com.google.dagger:dagger:2.8'</span>
<span class="o">}</span>
<span class="n">compileJava</span> <span class="o">{</span>
<span class="n">options</span><span class="o">.</span><span class="na">annotationProcessorPath</span> <span class="o">=</span> <span class="n">configurations</span><span class="o">.</span><span class="na">apt</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here, we’re creating a configuration, <code class="language-plaintext highlighter-rouge">apt</code>, that will contain all the annotation processors we use, and therefore also their specific transitive dependencies. Then we set the <code class="language-plaintext highlighter-rouge">annotationProcessorPath</code> to this configuration. What this enables is two-fold:</p>
<ul>
<li>it disables automatic annotation processor detection on the compile classpath, making the task start faster (faster up-to-date checks)</li>
<li>it will make use of the <code class="language-plaintext highlighter-rouge">processorpath</code> option of the Java compiler, and properly separate compile dependencies from the annotation processing path</li>
<li>it will enable <strong>compile avoidance</strong> : by explicitly saying that you use annotation processors, we can now make sure that everything that is found on classpath is only binary interfaces</li>
</ul>
<p>In particular, you will notice how <a href="https://google.github.io/dagger/">Dagger</a> cleanly separates its compiler from its annotations: we have <code class="language-plaintext highlighter-rouge">dagger-compiler</code> as an annotation processing dependency, and <code class="language-plaintext highlighter-rouge">dagger</code> (the annotations themselves) as <code class="language-plaintext highlighter-rouge">compile</code> dependencies. For Lombok, you would typically have to put the same dependency both in <code class="language-plaintext highlighter-rouge">compile</code> and <code class="language-plaintext highlighter-rouge">apt</code> to benefit from compile avoidance again.</p>
<p>However, some annotation processors do not separate these concerns properly and thus leak their implementation classes onto your classpath. Compile avoidance still works in this scenario: you need just put the jar on both the <code class="language-plaintext highlighter-rouge">apt</code> and <code class="language-plaintext highlighter-rouge">compileOnly</code> configurations.</p>
<h4 id="incremental-compile-with-annotation-processors">Incremental compile with annotation processors</h4>
<p>As said above, with annotation processors, Gradle does not know which files they are going to generate. Neither does it know where and based on what conditions. Therefore Grade disables the Java incremental compiler if annotation processors are in use, even if you declare them explicitly as we just have done. It is however possible to limit the impact of this to the set of classes that really use annotation processors. In short, you can declare a different source set, with a different compile task, that will use the annotation processor, and leave the other compile tasks without any kind of annotation processing: any change to a class that doesn’t use annotation processors would therefore benefit from incremental compilation, whereas any change to the sources that use annotations would trigger a full recompilation, but of that source set only. Here’s an example how to do it:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">configurations</span> <span class="o">{</span>
<span class="n">apt</span>
<span class="n">aptCompile</span>
<span class="o">}</span>
<span class="n">dependencies</span> <span class="o">{</span>
<span class="n">apt</span> <span class="s1">'com.google.dagger:dagger-compiler:2.8'</span>
<span class="n">aptCompile</span> <span class="s1">'com.google.dagger:dagger:2.8'</span>
<span class="o">}</span>
<span class="n">sourceSets</span> <span class="o">{</span>
<span class="n">processed</span> <span class="o">{</span>
<span class="n">java</span> <span class="o">{</span>
<span class="n">compileClasspath</span> <span class="o">+=</span> <span class="n">configurations</span><span class="o">.</span><span class="na">aptCompile</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">main</span> <span class="o">{</span>
<span class="n">java</span> <span class="o">{</span>
<span class="n">compileClasspath</span> <span class="o">+=</span> <span class="n">processed</span><span class="o">.</span><span class="na">output</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">compileProcessedJava</span> <span class="o">{</span>
<span class="n">options</span><span class="o">.</span><span class="na">annotationProcessorPath</span> <span class="o">=</span> <span class="n">configurations</span><span class="o">.</span><span class="na">apt</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In practice this may not be an easy split to perform, dependending on how much the <code class="language-plaintext highlighter-rouge">main</code> sources depend on classes found in the <code class="language-plaintext highlighter-rouge">processed</code> classes.
We are, however, exploring options to enable incremental compilation when annotation processors are present, which means that this shouldn’t be an issue in the future.</p>
<h2 id="java-libraries">Java libraries</h2>
<p>We at Gradle have been explaining for a long time why the Maven dependency model is broken, but it’s often hard to realize without a concrete example, because users just get used to the defect and deal with it as if it was natural. In particular, the <code class="language-plaintext highlighter-rouge">pom.xml</code> file is used both for building a component and for its publication metadata. Gradle has always worked differently, by having build scripts which are the “recipe” to build a component, and publications, which can be done to Maven, Ivy, or whatever other repositories you need to support. The publication contains metadata about how to consume the project, meaning that we clearly separate what you need to build a component from what you need as its consumer. Separating the two roles is extremely important, and it allows Gradle 3.4 to add a fundamental improvement to Java dependency management. There are multiple benefits you get with this new feature. One is better performance, as it complements the other performance features we have described above, but there are more.</p>
<h3 id="weve-all-been-doing-it-wrong">We’ve all been doing it wrong</h3>
<p>When building a Java project, there are two things being considered:</p>
<ul>
<li>what do I need to compile the project itself?</li>
<li>what do I need at runtime to execute the project?</li>
</ul>
<p>Which drives us naturally to declaring dependencies in two distinct scopes:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">compile</code> : the dependencies I need to compile the project</li>
<li><code class="language-plaintext highlighter-rouge">runtime</code> : the dependencies I need to run the project</li>
</ul>
<p>Maven and Gradle have both been using this for years. But since the beginning, we knew we were wrong. This view is over simplistic, because it doesn’t consider the <em>consumers</em> of your project. In particular, there are (at least) two kinds of projects in the Java world:</p>
<ul>
<li>applications, which are standalone, executable, and don’t expose any API</li>
<li>libraries, which are used by other libraries, or other applications, as bricks to build software, and therefore expose an API</li>
</ul>
<p>The problem with the simplistic approach of having two configurations (Gradle) or scopes (Maven) is that you don’t consider what is required in your API versus what is required by your implementation. In other words, you are <strong>leaking the compile dependencies of your component to downstream consumers</strong>.</p>
<p>Imagine that we are building an IoT application <code class="language-plaintext highlighter-rouge">home-automation</code> which depends on a <code class="language-plaintext highlighter-rouge">heat-sensor</code> library that has <code class="language-plaintext highlighter-rouge">commons-math3.jar</code> and <code class="language-plaintext highlighter-rouge">guava.jar</code> on its compile classpath. Then the compile classpath of <code class="language-plaintext highlighter-rouge">home-automation</code> will include <code class="language-plaintext highlighter-rouge">commons-math3.jar</code> and <code class="language-plaintext highlighter-rouge">guava.jar</code>. There are several consequences to this:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">home-automation</code> may start using classes from <code class="language-plaintext highlighter-rouge">commons-math3.jar</code> or <code class="language-plaintext highlighter-rouge">guava.jar</code> without really realizing they are transitive dependencies of <code class="language-plaintext highlighter-rouge">heat-sensor</code> (transitive dependency leakage).</li>
<li>the compile classpath of <code class="language-plaintext highlighter-rouge">home-automation</code> is bigger:
<ul>
<li>this increases the time spend on dependency resolution, up-to-date checking, classpath analysis and <code class="language-plaintext highlighter-rouge">javac</code>.</li>
<li>the new Gradle compile avoidance will be less efficient because changes in the classpath are more likely to happen and compile avoidance will not kick in. Specially, when you are using annotation processors where Gradle incremental compile is disabled, this comes with a high cost.</li>
</ul>
</li>
<li>you are increasing the chances of dependency hell (different versions of the same dependency on classpath)</li>
</ul>
<p>But the worst issue is that if the usage of <code class="language-plaintext highlighter-rouge">guava.jar</code> is a purely internal detail for <code class="language-plaintext highlighter-rouge">heat-sensor</code>, and that <code class="language-plaintext highlighter-rouge">home-automation</code> starts using it because it was found on classpath, then it becomes very hard to evolve <code class="language-plaintext highlighter-rouge">heat-sensor</code> because it would break consumers. The leakage of dependencies is a dreaded issue that leads to slowly evolving software and feature freeze, for the sake of backwards compatibility.</p>
<p>We know we’ve been doing this wrong, it’s time to fix it, and introduce the new Java Library plugin!</p>
<h3 id="introducing-the-java-library-plugin">Introducing the Java Library plugin</h3>
<p>Starting from Gradle 3.4, if you build a Java library, that is to say a component aimed at being consumed by other components (a component that is a dependency of another), then you should use the new Java Library plugin. Instead of writing:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">apply</span> <span class="nl">plugin:</span> <span class="s1">'java'</span>
</code></pre></div></div>
<p>use:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">apply</span> <span class="nl">plugin:</span> <span class="s1">'java-library'</span>
</code></pre></div></div>
<p>They both share a common infrastructure, but the <code class="language-plaintext highlighter-rouge">java-library</code> plugin exposes the concept of an API. Let’s migrate our <code class="language-plaintext highlighter-rouge">heat-sensor</code> library, which itself has 2 dependencies:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dependencies</span> <span class="o">{</span>
<span class="n">compile</span> <span class="s1">'org.apache.commons:commons-math3:3.6.1'</span>
<span class="n">compile</span> <span class="s1">'com.google.guava:guava:21.0'</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When you study the code in <code class="language-plaintext highlighter-rouge">heat-sensor</code>, you understand that <code class="language-plaintext highlighter-rouge">commons-math3</code> is exposed in the public API, while <code class="language-plaintext highlighter-rouge">guava</code> is purely internal:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">com.google.common.collect.Lists</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.commons.math3.stat.descriptive.SummaryStatistics</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">HeatSensor</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">SummaryStatistics</span> <span class="nf">getMeasures</span><span class="o">(</span><span class="kt">int</span> <span class="n">lastHours</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">List</span><span class="o"><</span><span class="nc">Measurement</span><span class="o">></span> <span class="n">measures</span> <span class="o">=</span> <span class="nc">Lists</span><span class="o">.</span><span class="na">newArrayList</span><span class="o">();</span> <span class="c1">// Google Guava is used internally, but doesn't leak into the public API</span>
<span class="c1">// ...</span>
<span class="k">return</span> <span class="n">stats</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>It means that if tomorrow, <code class="language-plaintext highlighter-rouge">heat-sensor</code> wants to switch from Guava to another collections library, it can do it without any impact to its consumers. But in practice, it’s only possible if we cleanly separate those dependencies into 2 buckets:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dependencies</span> <span class="o">{</span>
<span class="n">api</span> <span class="s1">'org.apache.commons:commons-math3:3.6.1'</span>
<span class="n">implementation</span> <span class="s1">'com.google.guava:guava:21.0'</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">api</code> bucket is used to declare dependencies that should transitively be visible by downstream consumers when they are compiled. The <code class="language-plaintext highlighter-rouge">implementation</code> bucket is used to declare dependencies which should not leak into the compile classpath of consumers (because they are purely internal details).</p>
<p>Now, when a consumer of <code class="language-plaintext highlighter-rouge">heat-sensor</code> is going to be compiled, it will <strong>only</strong> find <code class="language-plaintext highlighter-rouge">commons-math3.jar</code> on compile classpath, not <code class="language-plaintext highlighter-rouge">guava.jar</code>. So if <code class="language-plaintext highlighter-rouge">home-automation</code> accidently tries to use a class from Google Guava, it will fail at compile time, and the consumer needs to decide whether it really wants to introduce Guava as a dependency or not. On the other hand, if it tries to use a class from Apache Math3, which is an API dependency, then will succeed, because API dependencies are absolutely required at compile time.</p>
<h3 id="better-poms-than-maven">Better POMs than Maven</h3>
<p>So when does <code class="language-plaintext highlighter-rouge">implementation</code> matter? It matters at runtime only! This is why, now, the <code class="language-plaintext highlighter-rouge">pom.xml</code> file that Gradle generates whenever you choose to publish on a Maven repository is cleaner than what Maven can offer! Let’s look at what we generate for <code class="language-plaintext highlighter-rouge">heat-sensor</code>, using the <code class="language-plaintext highlighter-rouge">maven-publish</code> plugin:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="nt"><project</span> <span class="na">xsi:schemaLocation=</span><span class="s">"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"</span> <span class="na">xmlns=</span><span class="s">"http://maven.apache.org/POM/4.0.0"</span>
<span class="na">xmlns:xsi=</span><span class="s">"http://www.w3.org/2001/XMLSchema-instance"</span><span class="nt">></span>
<span class="nt"><modelVersion></span>4.0.0<span class="nt"></modelVersion></span>
<span class="nt"><groupId></span>com.acme<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>heat-sensor<span class="nt"></artifactId></span>
<span class="nt"><version></span>1.0.0-SNAPSHOT<span class="nt"></version></span>
<span class="nt"><dependencies></span>
<span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.apache.commons<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>commons-math3<span class="nt"></artifactId></span>
<span class="nt"><version></span>3.6.1<span class="nt"></version></span>
<span class="nt"><scope></span>compile<span class="nt"></scope></span>
<span class="nt"></dependency></span>
<span class="nt"><dependency></span>
<span class="nt"><groupId></span>com.google.guava<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>guava<span class="nt"></artifactId></span>
<span class="nt"><version></span>21.0<span class="nt"></version></span>
<span class="nt"><scope></span>runtime<span class="nt"></scope></span>
<span class="nt"></dependency></span>
<span class="nt"></dependencies></span>
<span class="nt"></project></span>
</code></pre></div></div>
<p>What you see is the <code class="language-plaintext highlighter-rouge">pom.xml</code> file that is published, and therefore <em>used by consumers</em>. And what does it say?</p>
<ul>
<li>to <em>compile</em> against <code class="language-plaintext highlighter-rouge">heat-sensor</code>, you need <code class="language-plaintext highlighter-rouge">commons-math3</code> on <code class="language-plaintext highlighter-rouge">compile</code> classpath</li>
<li>to <em>run</em> against <code class="language-plaintext highlighter-rouge">heat-sensor</code>, you need <code class="language-plaintext highlighter-rouge">guava</code> on <code class="language-plaintext highlighter-rouge">runtime</code> classpath</li>
</ul>
<p>This is <strong>very</strong> different from having the same <code class="language-plaintext highlighter-rouge">pom.xml</code> for both compiling the component and consuming it. Because to compile <code class="language-plaintext highlighter-rouge">heat-sensor</code> itself, you would need <code class="language-plaintext highlighter-rouge">guava</code> in <code class="language-plaintext highlighter-rouge">compile</code>. In short: Gradle generates better POM files than Maven, because it makes the difference between the producer and the consumer.</p>
<h3 id="more-uses-cases-more-configurations">More uses cases, more configurations</h3>
<p>You might be aware of the <code class="language-plaintext highlighter-rouge">compileOnly</code> configuration that <a href="https://blog.gradle.org/introducing-compile-only-dependencies">was introduced in Gradle 2.12</a>, which can be used to declare dependencies which are only required when compiling a component, but not at runtime (a typical use case is libraries which are embedded into a fat jar or <a href="https://github.com/johnrengelman/shadow">shadowed</a>). The <code class="language-plaintext highlighter-rouge">java-library</code> plugin provides a smooth migration path from the <code class="language-plaintext highlighter-rouge">java</code> plugin: if you are building an application, you can continue to use the <code class="language-plaintext highlighter-rouge">java</code> plugin. Otherwise, if it’s a library, just use the <code class="language-plaintext highlighter-rouge">java-library</code> plugin. But in both cases:</p>
<ul>
<li>instead of the <code class="language-plaintext highlighter-rouge">compile</code> configuration, you should use <code class="language-plaintext highlighter-rouge">implementation</code> instead</li>
<li>instead of the <code class="language-plaintext highlighter-rouge">runtime</code> configuration, you should use <code class="language-plaintext highlighter-rouge">runtimeOnly</code> configuration to declare dependencies which should only be visible at runtime</li>
<li>to resolve the runtime of a component, use <code class="language-plaintext highlighter-rouge">runtimeClasspath</code> instead of <code class="language-plaintext highlighter-rouge">runtime</code>.</li>
</ul>
<h3 id="impact-on-performance">Impact on performance</h3>
<p>To show you what the impact on performance can be, we added a benchmark which compares two scenarios:</p>
<ul>
<li>making an ABI-compatible change in a library, then recompile</li>
<li>making an ABI-incompatible change in a library, then recompile</li>
</ul>
<p>Only Gradle 3.4 supports the concept of library, and therefore uses the Java Library Plugin. And to make it even clearer, this benchmark does <em>not</em> use the incremental compiler (which would make things even faster, updates would almost be a no-op):</p>
<style>
#chart2 {
width: 100%
}
</style>
<div id="chart2"></div>
<script>
var chart2 = c3.generate({
bindto: '#chart2',
data: {
rows: [
['Scenario','Maven 3.3.9','Gradle 3.3','Gradle 3.4'],
['Library ABI-breaking change',10.60,3.80,2.60],
['Library ABI-compatible change',10.80,3.90,1.50]],
type: 'bar',
x: 'Scenario',
labels: true
},
axis: {
x: {
type: 'category'
},
y: {
label: 'seconds'
}
},
legend: {
position: 'right'
}
});
</script>
<p>As you can see, in addition to better modelling, there’s a strong impact on performance!</p>
<h2 id="conclusion">Conclusion</h2>
<p>Gradle 3.4 brings dramatic improvements to the Java ecosystem. Better incremental compilation and compile avoidance will significantly improve your productivity, while clean separation of API and implementation dependencies will avoid accidental leakage of dependencies and help you better model your software. Note that we have more goodness to come. In particular, separation of API and implementation is key to Java 9 success, with the awakening of <a href="http://openjdk.java.net/projects/jigsaw/">Project Jigsaw</a>. We’re going to add a way to declare what <em>packages</em> belong to your API, making it even closer to what Jigsaw will offer, but supported on older JDKs too.</p>
<p>In addition, Gradle 4.0 will ship with a build cache, which will strongly benefit from the improvements described in this post: it’s a mechanism which allows reusing, and sharing, the result of execution of tasks on a local machine or over the network. Typical use cases include switching branches, or simply checking out a project which has already been built by a colleague or on CI. Said differently, if you, or someone else, has already built something you need, you would get it from the cache instead of having to build it locally. For this, the build cache needs to generate a cache key which is, for java compile task, typically sensitive to the compile classpath. The improvements that ship in 3.4 will make this cache key more likely to be hit, because we would ignore what is not relevant to consumers (only ABI matters).</p>
<p>We encourage you to upgrade now, take a look at the <a href="https://docs.gradle.org/current/userguide/java_library_plugin.html">documentation of the new Java Library plugin</a> and discover all it can do for you!</p>
Announcing Buildship 2.02017-01-11T00:00:00-05:00https://blog.gradle.org/announcing-buildship-2.0Donat Csikos
<p>We are pleased to announce that version 2.0 of Buildship—our official Gradle support for Eclipse—is now available via the Eclipse Marketplace. This release adds <strong>support for composite builds</strong>, greatly reducing development turnaround time. <strong>The UI has been redesigned</strong> based on plenty of community feedback during the 1.x line. <strong>Project synchronization is now more accurate</strong> and <strong>project import requires one less step</strong>. We’ve added <strong>support for Gradle’s offline mode</strong> (thanks <a href="https://github.com/rlagoue">Rodrigue</a>!), and last but not least, third-party integrators can take advantage of our <strong>new <code class="language-plaintext highlighter-rouge">InvocationCustomizer</code> extension point</strong>. Read on for details about each of these new features.</p>
<h2 id="composite-build-support">Composite build support</h2>
<h4 id="what-is-a-composite-build">What is a composite build?</h4>
<p>The <a href="https://docs.gradle.org/current/userguide/composite_builds.html">composite build</a> feature in Gradle allows you to handle several distinct Gradle builds as if they were one big multi-project build. This dramatically shortens the turnaround time when you need to work on several projects that are normally developed separately.</p>
<p>Let’s assume you have written a Java library <code class="language-plaintext highlighter-rouge">lib</code>, used by many of your applications. You find a bug which only manifests itself in the <code class="language-plaintext highlighter-rouge">special-app</code>. The traditional development workflow would be to change some code in <code class="language-plaintext highlighter-rouge">lib</code> and install a snapshot into the local repository. Then you would have to change the build script of <code class="language-plaintext highlighter-rouge">special-app</code> to use that new snapshot and check if the bug is actually fixed.</p>
<!--break-->
<p>With composite builds, however, you can tell Gradle to treat both of these projects as one. This will let <code class="language-plaintext highlighter-rouge">special-app</code> depend directly on the output of the <code class="language-plaintext highlighter-rouge">lib</code> project.</p>
<p>You can learn more about composite builds in <a href="/introducing-composite-builds">this introductory blog post</a>.</p>
<h4 id="composite-builds-in-the-ide">Composite builds in the IDE</h4>
<p>If you develop <code class="language-plaintext highlighter-rouge">special-app</code> you probably have it imported in Eclipse with <code class="language-plaintext highlighter-rouge">lib</code> referenced as a binary dependency.</p>
<p><img src="/images/buildship-2.0-workspace.png" alt="Buildship workspace" /></p>
<p>There is not much difference between working with composite builds at the command line and working with them within Eclipse. To include <code class="language-plaintext highlighter-rouge">lib</code> you need only add an entry to your <code class="language-plaintext highlighter-rouge">settings.gradle</code> file, telling Gradle from which folder the additional build should be included.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">includeBuild</span> <span class="s1">'../lib'</span>
</code></pre></div></div>
<p>Then, to apply the changes right-click on the project and select <code class="language-plaintext highlighter-rouge">Gradle > Refresh Project</code>. After the synchronization finishes, you should see two things: the project from the included build is imported and the binary dependency is replaced with a project dependency.</p>
<p><img src="/images/buildship-2.0-workspace-with-composite.png" alt="Imported composite" /></p>
<p>Now, you can make changes to both projects with the benefit of complete IDE support: error markers, code completion, refactoring and more. Also, if you execute Gradle tests—or any other build task—from the <code class="language-plaintext highlighter-rouge">Gradle Tasks</code> view, the execution considers changes from the included builds.</p>
<p><img src="/images/buildship-2.0-task-execution.png" alt="Task execution with included builds" /></p>
<h4 id="limitations">Limitations</h4>
<p>When using composite builds from the IDE you should be aware of the following limitations:</p>
<ul>
<li>Composite builds support only works if the imported project uses Gradle 3.3 or above.</li>
<li>Task execution is disabled on included builds due to a <a href="https://docs.gradle.org/current/userguide/composite_builds.html#current_limitations_and_future_work">task addressing limitation in Gradle</a>.</li>
<li>Including WTP projects into a composite is not supported.</li>
</ul>
<h2 id="design-overhaul">Design overhaul</h2>
<p>We updated the Buildship user interface to align it with current Gradle branding as well as with the Eclipse design guidelines. The icons are now distinguishable by color-blind people and work well with Eclipse’s dark theme. Finally, high-definition images have been put in place for use with High-DPI displays.</p>
<p><img src="/images/buildship-2.0-design-overhaul.png" alt="New Buildship design" /></p>
<h2 id="import-wizard-simplification">Import wizard simplification</h2>
<p>We removed JAVA_HOME, program arguments, and JVM arguments configuration from the import and new project wizards. Users can still configure these properties via the <a href="https://docs.gradle.org/current/userguide/build_environment.html"><code class="language-plaintext highlighter-rouge">gradle.properties</code> file</a>.</p>
<h2 id="more-accurate-project-synchronization">More accurate project synchronization</h2>
<p>In Buildship 1.x if the project being imported had Eclipse descriptors then a dialog was shown to determine if those descriptors should be updated or deleted. This behavior was error-prone and distracting for users.</p>
<p>To avoid showing a dialog, we improved the project synchronization algorithm the following way: If the Gradle version used by the project can provide a specific attribute (e.g. project natures), it is completely overwritten. Manual modifications are only kept if Gradle provides no information about that attribute. This allows users of older Gradle versions to work around missing information in the model, while giving users of new Gradle versions a much more consistent experience.</p>
<h2 id="offline-mode-support">Offline mode support</h2>
<p>Users can now set Buildship to work offline via the workspace preferences. Once enabled, all Gradle invocations will receive an extra <code class="language-plaintext highlighter-rouge">--offline</code> argument.</p>
<p><img src="/images/buildship-2.0-offline-mode.png" alt="Offline mode support in preferences" /></p>
<h2 id="invocationcustomizer-extension-point">InvocationCustomizer extension point</h2>
<p>The <code class="language-plaintext highlighter-rouge">InvocationCustomizer</code> extension point enables Eclipse plugins to contribute extra arguments to Gradle builds. This allows integrators to add init scripts or control project properties from the IDE. For a sample implementation check out the <a href="https://github.com/eclipse/buildship/blob/master/docs/development/ApiExamples.md">Buildship documentation</a>.</p>
<h2 id="breaking-changes">Breaking changes</h2>
<p>This release introduces the following breaking changes:</p>
<ul>
<li>Minimum Java version set to 1.7</li>
<li>Minimum Eclipse version is set to 4.2</li>
<li>Project renaming is disabled for projects that are located directly under the Eclipse workspace location.</li>
<li>Projects migrating from Eclipse Mars directly to Eclipse Oxygen need to be reimported</li>
<li>JAVA_HOME can no longer be configured on import, please use <code class="language-plaintext highlighter-rouge">gradle.properties</code> instead</li>
<li>Java arguments and Gradle properties can no longer be configured on import, please use <code class="language-plaintext highlighter-rouge">gradle.properties</code> instead</li>
</ul>
<h2 id="installation">Installation</h2>
<p>Buildship 2.0 is available from the <a href="https://marketplace.eclipse.org/content/buildship-gradle-integration">Eclipse Marketplace</a> or from the <a href="https://github.com/eclipse/buildship/blob/master/docs/user/Installation.md#installing-from-eclipseorg-update-site">eclipse.org</a> update sites. Please note that the update site URL has changed therefore no automatic update is available for this release.</p>
Custom Data in Build Scans2017-01-04T00:00:00-05:00https://blog.gradle.org/custom-data-in-build-scansCraig Atkinson
<p>Build scans are a great way to easily share data about your build, but what if your team wants to add their own data to those build scans? They can! In addition to the extensive information automatically captured in build scans, you can attach your own custom data to provide even deeper insights into your build. This custom data can take the form of tags, links, and arbitrary custom values in a key-value format.</p>
<p>By adding custom data to your build scans you can make it easy to find builds of a certain type, give quick links to the applicable source code commit on GitHub, add helpful CI build information, and much more. Then, when you share the single build scan link with a teammate, they get quick and easy access to a plethora of information about your build, making it easier for them to diagnose build environment issues, fix test failures, and so on.</p>
<p>If build scans are new to you, you can learn about them in our <a href="https://blog.gradle.org/introducing-build-scans">introductory blog post on the topic</a>. You can also find more details in the <a href="https://docs.gradle.com/scans/">Build Scans User Manual</a>, <a href="https://gradle.com/#explore">explore some example build scans</a> or <a href="https://github.com/gradle/gradle-build-scan-quickstart">experiment with this sample build scan project</a>.</p>
<p>Now let’s go through some examples of adding custom data into your build scans (see the user manual for <a href="https://docs.gradle.com/scans/#extending_build_scans">additional examples</a>).</p>
<!--break-->
<h2 id="tags">Tags</h2>
<p>Let’s start with the simplest type of custom data: tags. Tags are a way to add simple pieces of metadata to your build scan. You can use tags to add context to your build, such as whether the build was run locally or on a CI server, whether the build had any local changes, the error type of a failing build, etc. Here is an <a href="https://scans.gradle.com/s/iodpryfwszmby">example build scan</a> that tags the build as having:</p>
<ul>
<li>run on CI</li>
<li>come from the master branch</li>
<li>included local code changes (“dirty”)</li>
</ul>
<p>For example, to attach a tag showing whether the build ran locally or on a CI server, you can add the following to its build script:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(</span><span class="n">System</span><span class="o">.</span><span class="na">getenv</span><span class="o">(</span><span class="s2">"CI"</span><span class="o">))</span> <span class="o">{</span>
<span class="n">buildScan</span><span class="o">.</span><span class="na">tag</span> <span class="s2">"CI"</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="n">buildScan</span><span class="o">.</span><span class="na">tag</span> <span class="s2">"LOCAL"</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The tag is then displayed under the project name when viewing the build scan:</p>
<p><img src="/images/build-scan-ui-tag.png" alt="Build scan tag" /></p>
<h2 id="links">Links</h2>
<p>In addition to tags, you can include links that readers of your build scan might find useful. For example, you could include a convenient link to the project source on GitHub or a link to the CI results of the Gradle build. This <a href="https://scans.gradle.com/s/gtoxe7hyq3bl6">example build scan</a> demonstrates what such links look like.</p>
<p>Let’s say your CI tool makes the build results URL available as an environment variable. You could grab that value and add it as a custom link by using the following code in your build script:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(</span><span class="n">System</span><span class="o">.</span><span class="na">getenv</span><span class="o">(</span><span class="s2">"CI"</span><span class="o">))</span> <span class="o">{</span>
<span class="n">buildScan</span><span class="o">.</span><span class="na">link</span> <span class="s2">"CI build"</span><span class="o">,</span> <span class="n">System</span><span class="o">.</span><span class="na">getenv</span><span class="o">(</span><span class="s2">"BUILD_URL"</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>You also have the flexibility to add a link to the current revision or commit of the project’s source code. The following example links the build scan to its corresponding commit on GitHub (as long as the Git command line tools are available):</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">String</span> <span class="n">commitId</span> <span class="o">=</span> <span class="s1">'git rev-parse --verify HEAD'</span><span class="o">.</span><span class="na">execute</span><span class="o">().</span><span class="na">text</span><span class="o">.</span><span class="na">trim</span><span class="o">()</span>
<span class="n">buildScan</span><span class="o">.</span><span class="na">link</span> <span class="s2">"Source"</span><span class="o">,</span> <span class="s2">"https://github.com/gradle/gradle-build-scan-quickstart/tree/"</span> <span class="o">+</span> <span class="n">commitId</span>
</code></pre></div></div>
<p>Links are displayed in the top section when viewing the build scan:</p>
<p><img src="/images/build-scan-ui-links.png" alt="Build scan links" /></p>
<h2 id="custom-values">Custom values</h2>
<p>Custom values can be used to make <em>any</em> information part of the build scan. In <a href="https://scans.gradle.com/s/nadne7cvexins#custom-values">this example build scan</a>, you can see the corresponding CI build date, CI build number and the name of the Git branch as custom values. These values are available when viewing the build scan or when searching for build scans in <a href="https://gradle.com/enterprise">Gradle Enterprise</a>. Let’s go through a couple of examples showing how you can add custom values to your build scan.</p>
<p>In our first example, we assume your CI tool injects build information into the build via environment variables. You could then use the following code in your build script to attach the build number and date to the build scan:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(</span><span class="n">System</span><span class="o">.</span><span class="na">getenv</span><span class="o">(</span><span class="s2">"CI"</span><span class="o">))</span> <span class="o">{</span>
<span class="n">buildScan</span><span class="o">.</span><span class="na">value</span> <span class="s2">"CI build number"</span><span class="o">,</span> <span class="n">System</span><span class="o">.</span><span class="na">getenv</span><span class="o">(</span><span class="s2">"BUILD_NUMBER"</span><span class="o">)</span>
<span class="n">buildScan</span><span class="o">.</span><span class="na">value</span> <span class="s2">"CI build date"</span><span class="o">,</span> <span class="n">System</span><span class="o">.</span><span class="na">getenv</span><span class="o">(</span><span class="s2">"BUILD_DATE"</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Since we are setting these custom values from inside a Gradle build script, you have the power to do things like run external commands to capture more information about the project status. For example, you could add the current Git branch of the build by running a Git command and setting a custom value with the result:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">String</span> <span class="n">branchName</span> <span class="o">=</span> <span class="s1">'git rev-parse --abbrev-ref HEAD'</span><span class="o">.</span><span class="na">execute</span><span class="o">().</span><span class="na">text</span><span class="o">.</span><span class="na">trim</span><span class="o">()</span>
<span class="n">buildScan</span><span class="o">.</span><span class="na">value</span> <span class="s2">"Git branch"</span><span class="o">,</span> <span class="n">branchName</span>
</code></pre></div></div>
<p>The custom values are displayed on the main page when viewing the build scan:</p>
<p><img src="/images/build-scan-ui-values.png" alt="Build scan custom values" /></p>
<h2 id="command-line">Command line</h2>
<p>To give you greater flexibility in how you pass custom data to your build scan, you can also specify tags, links, and custom values on the command line. For example, you can quickly attach ad-hoc information to your build scan in order to:</p>
<ul>
<li>help debug a specific local build failure</li>
<li>tag an experimental build</li>
<li>add CI-specific custom data without modifying your build script</li>
</ul>
<p>You do this by specifying system properties with the appropriate names, as demonstrated by these examples:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ gradle build -Dscan.tag.EXPERIMENTAL
$ gradle build -Dscan.link.buildUrl=$CI_BUILD_URL
$ gradle build -Dscan.value.buildNumber=$CI_BUILD_NUMBER
</code></pre></div></div>
<p>The first adds a tag named “EXPERIMENTAL”, the second adds a link titled “buildUrl”, and the third adds a custom value called “buildNumber”.</p>
<h2 id="searching-based-on-custom-data">Searching based on custom data</h2>
<p>When using build scans on-premises with <a href="https://gradle.com/enterprise">Gradle Enterprise</a>, you can search for build scans based on custom data such as tags and custom values. For example, you can search for all builds that ran on CI against the master branch using the terms shown in this screenshot:</p>
<p><img src="/images/build-scan-ui-search-results.png" alt="Build scan serch results" /></p>
<h2 id="live-demo">Live Demo</h2>
<p>For a live demo of adding custom data with even more examples, check out this recent talk by Luke Daley and Etienne Studer at the Bay Area Gradle Users meetup. The video starts with an overview of build scans and dives into the details of adding custom data around the 22:30 mark.</p>
<iframe class="fullwidth" width="640" height="360" src="https://www.youtube.com/embed/1-rjqhLcyYM" frameborder="0" allowfullscreen=""></iframe>
<p>Adding custom data to your build scans gives you the power and flexibility to augment your build scans with tags, links, or other data tailored to your team’s specific needs. Then you have even more information available to easily share with your teammates in a build scan—reducing the guesswork of diagnosing build failures. Happy scanning!</p>
Save the Date: Free Gradle Training in January2016-12-08T00:00:00-05:00https://blog.gradle.org/january-free-trainingSchalk W. Cronjé
<p>Getting started with a new technology can be daunting. Learning the basics by reading manuals and blog posts and searching forums can be time-consuming. And getting a whole team up to speed is a challenge all its own. That’s why for years now, we’ve offered a range of Gradle training courses to help teams fast-track the process of learning and mastering Gradle.</p>
<p>Our flagship <em>Introduction to Gradle</em> course has always been our most popular, and in October we ran an experiment with it: we gave it away for free. That’s a steep discount from the usual price of $900 per seat, but we wanted to see just how many people we could help to learn Gradle if cost were not a factor.</p>
<p>We’re happy to report that this first free training was an overwhelming success, and are even happier to announce that we’ll offer a second free Introduction to Gradle training on January 11th and 12th. The course will be led by Gradle veterans <a href="https://github.com/ghale">Gary Hale</a> and <a href="https://github.com/bmuschko">Ben Muschko</a> over these two consecutive, 4-hour training days.</p>
<p>In it, you and your colleagues will get everything you need to start creating and running your own Gradle builds with confidence. There will be plenty of hands-on labs and opportunies for Q&A with the instructors. We look forward to seeing you there!</p>
<p><strong>Details</strong>:</p>
<ul>
<li><strong>When?</strong> 8:30am–12:30am PST on January 11–12, 2017</li>
<li><strong>Where?</strong> Online via Zoom webinar - <a href="https://go.gradle.com/free-training">Register Now</a></li>
</ul>
November 15th Bay Area Gradle Users Meetup: Recap and Videos2016-11-22T00:00:00-05:00https://blog.gradle.org/november-bay-area-meetup-recapPeter Ledbrook
<p>We’d like to thank everyone that came along to our <a href="/november-15th-bay-area-gradle-users-meetup">Bay Area Gradle Users meetup</a> last week, and we’d like to thank LinkedIn once again for hosting us—it was a great event! For those who were unable to attend for reasons of distance, time or anything else, we filmed both sessions and are delighted to make the videos available to everyone.</p>
<p>As described in that earlier blog post, Szczepan Faber and Hans Dockter talked in detail about Gradle’s new composite build feature:</p>
<iframe class="fullwidth" width="640" height="360" src="https://www.youtube.com/embed/krv317ZOWGg" frameborder="0" allowfullscreen=""></iframe>
<p>In particular, Szczepan demonstrated the potential for working with multi-repository projects in an IDE as if they were part of the same multi-project build. You’ll find that in the first 10 minutes of the video.</p>
<p>That talk was followed by Luke Daley and Etienne Studer presenting the advantages of using custom data in your build scans:</p>
<iframe class="fullwidth" width="640" height="360" src="https://www.youtube.com/embed/1-rjqhLcyYM" frameborder="0" allowfullscreen=""></iframe>
<p>It’s well worth setting aside some time to watch both of these if you can.</p>
<p>We hope you find these videos useful and we look forward to seeing many of you at the next meetup!</p>
Webinar: Customizing Build Scan Data2016-11-02T08:00:00-04:00https://blog.gradle.org/webinar-customizing-build-scan-dataPeter Ledbrook
<p>For those of you who can’t make our <a href="/november-15th-bay-area-gradle-users-meetup">Bay Area meetup on November 15th</a>, we’re putting on a webinar a couple days later that will cover one of the same topics: <em>customizing build scan data</em>. Even better, the webinar will be delivered by one of Gradle’s best: <a href="https://github.com/mark-vieira">Mark Vieira</a>! So come join us for a half-hour of valuable learning and discover how to maximize the benefit of your build scans.</p>
<p><strong>Details</strong>:</p>
<ul>
<li><strong>When?</strong> 11:00 - 11:30am PT on November 17, 2016</li>
<li><strong>Where?</strong> Online via Zoom webinar - <a href="https://go.gradle.com/november-webinar/">Register Now</a></li>
</ul>
<p>There will be an opportunity for Q&A and a recording of the webinar will be made available to attendees. We hope you’ll join us!</p>
November 15th Bay Area Gradle Users Meetup2016-11-02T07:00:00-04:00https://blog.gradle.org/november-15th-bay-area-gradle-users-meetupPeter Ledbrook
<p>Everyone has an opportunity to engage with the Gradle team online through a variety of channels, but nothing beats meeting people face to face. If you are around the Silicon Valley area on November 15th, you can meet three of the team at the <a href="http://www.meetup.com/Bay-Area-Gradle-Users">Bay Area Gradle Users</a> meetup along with an expert user and build master from LinkedIn.</p>
<p>We have two great talks lined up, the first of which introduces you to an exciting new feature within Gradle—composite builds—while the second shows you how to get more out of your build scans with custom data.</p>
<p><strong>Details:</strong></p>
<ul>
<li><strong>Who?</strong> Hans Dockter, Szczepan Faber (LinkedIn), Luke Daley, Etienne Studer</li>
<li><strong>When?</strong> 6:00pm PT on November 15, 2016</li>
<li><strong>Where?</strong> LinkedIn, 605 W. Maude, Sunnyvale, CA</li>
<li><strong>RSVP</strong>: If you plan to attend, please <a href="https://www.eventbrite.com/e/bay-area-gradle-users-meetup-tickets-29087446335">register beforehand via Eventbrite</a></li>
</ul>
<p>We hope to see you there!</p>
<h2 id="hans-dockter-and-szczepan-faber-on-composite-builds">Hans Dockter and Szczepan Faber on Composite Builds</h2>
<p>Many of you will be familiar with Gradle’s multi-project build support which allows you set up dependencies between projects, e.g. where the output of one project—say, a JAR file—is required by another. But this only works if the projects are part of the same multi-project build.</p>
<p>In this talk, Hans and Szczepan will explain how Gradle’s new support for <a href="/introducing-composite-builds">composite builds</a> extends this behavior to projects that are otherwise independent of one another. Want to test a local fix to a library one of your projects depends on? Now you can—without having to publish a new version of that library. Composite builds also enable structuring your projects in new ways, since you no longer need to keep all projects in one repository or directory hierarchy.</p>
<p>Hans won’t just talk about the topic, he’ll also show you how the feature works in practice. You’ll come away with a firm understanding of the value of composite builds and how you might put them to use in your own projects.</p>
<h2 id="luke-daley-and-etienne-studer-on-customizing-build-scan-data">Luke Daley and Etienne Studer on Customizing Build Scan Data</h2>
<p><a href="/introducing-build-scans">Build scans</a> already provide deep insights into your build by reporting key metrics and data. These are incredibly useful on their own, but Gradle is designed around the understanding that no two builds are the same. That’s why build scans allow you to add custom tags, links, and values from your builds. These custom annotations can help you bring out insights that are very specific to your build and to the environment in which your build is run.</p>
<p>In this talk, Luke and Etienne will show examples of how you can use this feature to add extra value to your build scans. The aim is to sow the seeds of inspiration for your own builds using a feature that you might otherwise overlook.</p>
The Road to Gradle Script Kotlin 1.02016-10-26T00:00:00-04:00https://blog.gradle.org/kotlin-scripting-updateRodrigo B. de Oliveira
<p>Five months ago we <a href="/kotlin-meets-gradle">announced the first pre-release of Gradle Script Kotlin</a>, and we thought now would be a good time to review the progress we’ve made since. We have shipped eight additional pre-releases during that time, and the road to 1.0 is looking clearer every day. So let’s take a look at the ground we’ve covered so far and where we’re going from here, shall we?</p>
<h2 id="v010">v0.1.0</h2>
<p>As you may recall, this is what our <a href="https://github.com/gradle/gradle-script-kotlin/blob/cc14d3/samples/hello-world/build.gradle.kts"><code class="language-plaintext highlighter-rouge">hello-world</code> sample</a> looked like at the time of our first release:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">org.gradle.api.plugins.*</span>
<span class="k">import</span> <span class="nn">org.gradle.script.lang.kotlin.*</span>
<span class="n">apply</span><span class="p"><</span><span class="nc">ApplicationPlugin</span><span class="p">>()</span>
<span class="n">configure</span><span class="p"><</span><span class="nc">ApplicationPluginConvention</span><span class="p">></span> <span class="p">{</span>
<span class="n">mainClassName</span> <span class="p">=</span> <span class="s">"samples.HelloWorld"</span>
<span class="p">}</span>
<span class="nf">repositories</span> <span class="p">{</span>
<span class="nf">jcenter</span><span class="p">()</span>
<span class="p">}</span>
<span class="nf">dependencies</span> <span class="p">{</span>
<span class="s">"testCompile"</span><span class="p">(</span><span class="s">"junit:junit:4.12"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<!--break-->
<p>Oh, that annoying <code class="language-plaintext highlighter-rouge">org.gradle.script.lang.kotlin.*</code> import! The publicly condemned, IDE unfriendly, string-based <code class="language-plaintext highlighter-rouge">"testCompile"</code> dependency configuration! And of course—for those souls brave enough to have tried them—the infamous <code class="language-plaintext highlighter-rouge">generateKtsConfig</code> and <code class="language-plaintext highlighter-rouge">patchIdeaConfig</code> tasks required to get Kotlin-based build scripts working in IDEA. These were early days, no doubt, and they brought with them a few rough edges.</p>
<p>But despite its flaws, the programming language and IDE experience in 0.1.0 was already so good it got us hooked. As for the rough edges, we could already see ways to smooth them out, which led to the release of <a href="https://github.com/gradle/gradle-script-kotlin/releases/tag/v0.2.0">0.2.0</a> one month later.</p>
<h2 id="v020">v0.2.0</h2>
<p>With <a href="https://github.com/gradle/gradle-script-kotlin/issues/33">implicit imports</a> and a <a href="https://github.com/gradle/gradle-script-kotlin/issues/36">tooling-friendly alternative to string-based dependency configurations</a>, <code class="language-plaintext highlighter-rouge">hello-world</code> 0.2.0 started looking clean and concise:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">apply</span><span class="p"><</span><span class="nc">ApplicationPlugin</span><span class="p">>()</span>
<span class="n">configure</span><span class="p"><</span><span class="nc">ApplicationPluginConvention</span><span class="p">></span> <span class="p">{</span>
<span class="n">mainClassName</span> <span class="p">=</span> <span class="s">"samples.HelloWorld"</span>
<span class="p">}</span>
<span class="nf">repositories</span> <span class="p">{</span>
<span class="nf">jcenter</span><span class="p">()</span>
<span class="p">}</span>
<span class="nf">dependencies</span> <span class="p">{</span>
<span class="nf">testCompile</span><span class="p">(</span><span class="s">"junit:junit:4.12"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a href="https://github.com/gradle/gradle-script-kotlin/issues/26">Seamless project imports</a> meant that Kotlin-based builds in IDEA started working out of the box, and the days of mistyping <code class="language-plaintext highlighter-rouge">generateKtsConfig</code> and <code class="language-plaintext highlighter-rouge">patchIdeaConfig</code> were no more.</p>
<p>Perhaps most importantly, 0.2.0’s <a href="https://github.com/gradle/gradle-script-kotlin/issues/29">support for build script dependencies and external plugins</a> made Gradle Script Kotlin a viable choice for many real-world projects.</p>
<h2 id="v030">v0.3.0</h2>
<p><a href="https://github.com/gradle/gradle-script-kotlin/releases/tag/v0.3.0">0.3.0</a> was a major milestone for the project, as it was the first version to be included in a production Gradle distribution—<a href="https://github.com/gradle/gradle/releases/tag/v3.0.0">Gradle 3.0</a>, no less!</p>
<p>And 0.3.0 was all about that <a href="https://kotlin.link/">Kotlin</a>! The <a href="https://blog.jetbrains.com/kotlin/2016/07/first-glimpse-of-kotlin-1-1-coroutines-type-aliases-and-more/">new Kotlin 1.1-M01 compiler</a>, support for <a href="https://github.com/gradle/gradle-script-kotlin/issues/84">Kotlin-based plugins</a> and <a href="https://github.com/gradle/gradle-script-kotlin/issues/86"><code class="language-plaintext highlighter-rouge">buildSrc</code> directories</a> plus some sugary <a href="https://github.com/gradle/gradle-script-kotlin/issues/103">Kotlin-Groovy interoperability</a> primitives:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gradle</span><span class="p">.</span><span class="nf">buildFinished</span><span class="p">(</span><span class="n">closureOf</span><span class="p"><</span><span class="nc">BuildResult</span><span class="p">></span> <span class="p">{</span>
<span class="nf">println</span><span class="p">(</span><span class="s">"$action finished"</span><span class="p">)</span> <span class="c1">// $action refers to BuildResult.getAction()</span>
<span class="p">})</span>
</code></pre></div></div>
<p>With Gradle 3.0 out the door, the #gradle channel of the public <a href="http://kotlinslackin.herokuapp.com/">Kotlin Slack</a> saw an increase in participation which helped us greatly in prioritizing the work that would come next.</p>
<h2 id="v031">v0.3.1</h2>
<p>We noticed people struggling with the lack of a more type-safe and IDE-friendly way of configuring dependencies, so <a href="https://github.com/gradle/gradle-script-kotlin/releases/tag/v0.3.1">0.3.1</a> came with a <a href="https://github.com/gradle/gradle-script-kotlin/issues/107">much-improved dependencies DSL</a>:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">dependencies</span> <span class="p">{</span>
<span class="nf">default</span><span class="p">(</span><span class="n">group</span> <span class="p">=</span> <span class="s">"org.gradle"</span><span class="p">,</span> <span class="n">name</span> <span class="p">=</span> <span class="s">"foo"</span><span class="p">,</span> <span class="n">version</span> <span class="p">=</span> <span class="s">"1.0"</span><span class="p">)</span> <span class="p">{</span>
<span class="n">isForce</span> <span class="p">=</span> <span class="k">true</span>
<span class="p">}</span>
<span class="nf">compile</span><span class="p">(</span><span class="n">group</span> <span class="p">=</span> <span class="s">"org.gradle"</span><span class="p">,</span> <span class="n">name</span> <span class="p">=</span> <span class="s">"bar"</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">exclude</span><span class="p">(</span><span class="n">module</span> <span class="p">=</span> <span class="s">"foo"</span><span class="p">)</span>
<span class="p">}</span>
<span class="nf">runtime</span><span class="p">(</span><span class="s">"org.gradle:baz:1.0-SNAPSHOT"</span><span class="p">)</span> <span class="p">{</span>
<span class="n">isChanging</span> <span class="p">=</span> <span class="k">true</span>
<span class="n">isTransitive</span> <span class="p">=</span> <span class="k">false</span>
<span class="p">}</span>
<span class="nf">testCompile</span><span class="p">(</span><span class="n">group</span> <span class="p">=</span> <span class="s">"junit"</span><span class="p">,</span> <span class="n">name</span> <span class="p">=</span> <span class="s">"junit"</span><span class="p">)</span>
<span class="nf">testRuntime</span><span class="p">(</span><span class="nf">project</span><span class="p">(</span><span class="n">path</span> <span class="p">=</span> <span class="s">":core"</span><span class="p">))</span> <span class="p">{</span>
<span class="nf">exclude</span><span class="p">(</span><span class="n">group</span> <span class="p">=</span> <span class="s">"org.gradle"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The <a href="https://github.com/gradle/gradle-script-kotlin/issues/108">upgrade to Kotlin 1.1-dev-2053</a> notably enhanced the performance of code assistance within IDEA and thanks to a <a href="https://github.com/tyvsmith">valuable member of the community</a> the first Gradle Script Kotlin <a href="https://github.com/gradle/gradle-script-kotlin/tree/96b6fe/samples/hello-android">Android sample</a> was published.</p>
<h2 id="v032">v0.3.2</h2>
<p>With <a href="https://github.com/gradle/gradle-script-kotlin/releases/tag/v0.3.2">0.3.2</a> we decided to tackle <a href="https://www.youtube.com/watch?v=vv4zh_oPBTw&feature=youtu.be&t=1387">the dreaded <code class="language-plaintext highlighter-rouge">it</code> problem</a> head-on via <a href="https://github.com/gradle/gradle-script-kotlin/issues/117">runtime code generation of Kotlin extensions</a>. What is the dreaded <code class="language-plaintext highlighter-rouge">it</code> problem? Take the use of <a href="https://docs.gradle.org/3.1/javadoc/org/gradle/api/Project.html#copySpec(org.gradle.api.Action)"><code class="language-plaintext highlighter-rouge">copySpec</code></a> as an example. Prior to 0.3.2, one would have written:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">copySpec</span> <span class="p">{</span>
<span class="n">it</span><span class="p">.</span><span class="nf">from</span><span class="p">(</span><span class="s">"src/data"</span><span class="p">)</span>
<span class="n">it</span><span class="p">.</span><span class="nf">include</span><span class="p">(</span><span class="s">"*.properties"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This syntax didn’t read very well, and was a departure from the fluid, readable DSL Gradle has long been known for. But never fear—with 0.3.2 <code class="language-plaintext highlighter-rouge">it</code> was gone:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">copySpec</span> <span class="p">{</span>
<span class="nf">from</span><span class="p">(</span><span class="s">"src/data"</span><span class="p">)</span>
<span class="nf">include</span><span class="p">(</span><span class="s">"*.properties"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="v033-and-v040">v0.3.3 and v0.4.0</h2>
<p>The recently-released versions <a href="https://github.com/gradle/gradle-script-kotlin/releases/tag/v0.3.3">0.3.3</a> and <a href="https://github.com/gradle/gradle-script-kotlin/releases/tag/v0.4.0">0.4.0</a> shipped the <a href="https://github.com/gradle/gradle-script-kotlin/issues/137">first</a> of a series of improvements to <a href="https://github.com/gradle/gradle-script-kotlin/issues/112">multi-project builds</a> including the ability to <a href="https://github.com/gradle/gradle-script-kotlin/blob/7c74044cd84c4c426f1bca9af9f48bf332620c73/samples/multi-project-with-buildSrc/README.md">define custom build logic using Kotlin in <code class="language-plaintext highlighter-rouge">buildSrc</code></a>.</p>
<p><a href="https://github.com/gradle/gradle-script-kotlin/releases/tag/v0.4.0">0.4.0</a> is available now and will ship with the forthcoming Gradle 3.2 distribution.</p>
<h2 id="toward-v100">Toward v1.0.0</h2>
<p>What’s next, you ask? Here are some of the highlights of our upcoming releases in three key areas:</p>
<ol>
<li><a href="https://github.com/gradle/gradle-script-kotlin/issues/160">Performance</a>: Faster project configuration via caching of compiled build scripts (<a href="https://github.com/gradle/gradle-script-kotlin/issues/31">#31</a>).</li>
<li><a href="https://github.com/gradle/gradle-script-kotlin/issues/54">Usability</a>: Type-safe accessors for extensions and conventions contributed by plugins (<a href="https://github.com/gradle/gradle-script-kotlin/issues/159">#159</a>); Comprehensive
documentation (<a href="https://github.com/gradle/gradle-script-kotlin/issues/106">#106</a>).</li>
<li><a href="https://github.com/gradle/gradle-script-kotlin/issues/30">Convenience</a>: Declarative and tooling-friendly application of plugins, a.k.a., the <code class="language-plaintext highlighter-rouge">plugins</code> block (<a href="https://github.com/gradle/gradle-script-kotlin/issues/168">#168</a>).</li>
</ol>
<p>Altogether, here’s how we envision the <code class="language-plaintext highlighter-rouge">hello-world</code> sample will look in Gradle Script Kotlin 1.0:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">plugins</span> <span class="p">{</span>
<span class="n">application</span>
<span class="p">}</span>
<span class="nf">application</span> <span class="p">{</span>
<span class="n">mainClassName</span> <span class="p">=</span> <span class="s">"samples.HelloWorld"</span>
<span class="p">}</span>
<span class="nf">repositories</span> <span class="p">{</span>
<span class="nf">jcenter</span><span class="p">()</span>
<span class="p">}</span>
<span class="nf">dependencies</span> <span class="p">{</span>
<span class="nf">testCompile</span><span class="p">(</span><span class="s">"junit:junit:4.12"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>How does that look to you? We’d love to hear what you think.</p>
<p>A big thanks to everyone that’s been along for the ride so far, and if you’re just getting started with Gradle Script Kotlin, welcome!</p>
Now Open: GitHub Issues for Gradle2016-10-19T00:00:00-04:00https://blog.gradle.org/github-issuesEric Wendelin
<p>Gradle has been an open-source project since its inception, but as a team we haven’t always lived up to the spirit of modern open-source collaboration. For example, we haven’t made it easy for folks to stay abreast of what we’re working on, and we haven’t had a clear and simple process for users to submit feature requests or bugs against a proper issue tracker.</p>
<p>We’re very happy to announce that all of that is changing today. We’ve opened up GitHub Issues on the Gradle repository, and what follows are the immediate, medium-, and long-term changes we’re making to put—and keep!—the needs of the Gradle community front and center.</p>
<!--break-->
<h2 id="effective-immediately">Effective Immediately</h2>
<p>✔︎ You can now submit feature requests and bugs through <a href="https://github.com/gradle/gradle/issues">GitHub Issues</a>. We’ve put together a simple set of guidelines in the form of an <a href="https://github.com/gradle/gradle/blob/master/.github/ISSUE_TEMPLATE.md">issue template</a> so you’ll be presented with them each time you submit a new issue. We’ll label issues that don’t follow these guidelines as <code class="language-plaintext highlighter-rouge">not-actionable</code> and we’ll add a comment asking for what’s missing. To keep things clean, we’ll close these issues after a week of inactivity.</p>
<p>✔︎ Actionable issues you submit will be prioritized alongside other Gradle improvements. Low-cost changes that benefit a large number of users are the easiest to justify, so please add a <a href="https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments">👍 reaction</a> to issues that matter to you. Like many other GitHub projects, we’ll use these reactions as a simple kind of voting system.</p>
<p>✔︎ We’ll manage issue priority and workflow using <a href="https://www.zenhub.io">ZenHub</a>. If you’re not already familiar, ZenHub adds a number of features to GitHub Issues through its clever browser extension. It allows us to group related issues into <a href="https://www.zenhub.com/blog/working-with-epics-in-github/">epics</a> and to visualize everything via a kanban board. Installing and using ZenHub is by no means a requirement, but do check it out if you’d like to get an additional level of insight about what we’re working on.</p>
<p>✔︎ Pull requests will be acknowledged within a week and reviews of them will be done through GitHub’s built-in <a href="https://www.youtube.com/watch?v=HW0RPaJqm4g">pull request reviews</a>. Please continue to follow the <a href="https://github.com/gradle/gradle/blob/master/CONTRIBUTING.md">contribution guidelines</a>—in a project as large and widely-used as Gradle, it’s crucial to be rigorous and get things right.</p>
<h2 id="medium-term-transition">Medium-Term Transition</h2>
<p>Open <a href="https://issues.gradle.org/">JIRA</a> issues will be migrated to GitHub, and no new issues will be put into JIRA after the release of Gradle 3.2. We’ll ask for examples and/or clarification on JIRA issues that are not actionable so that they can be prioritized.</p>
<p>To draw a clearer roadmap, we’ll add and maintain high-level feature <a href="https://github.com/gradle/gradle/labels/epic">epics</a> and mark them with milestones representing future Gradle releases.</p>
<p>Our <a href="https://discuss.gradle.org/c/bugs">bugs forum</a> will no longer be necessary and will be made read-only. For general usage questions and potential bugs for which no reproducible test case or <a href="/introducing-build-scans">build scan</a> can be provided, please continue to use the <a href="https://discuss.gradle.org/c/help-discuss">help/discuss forum</a>.</p>
<h2 id="long-term-unity">Long-Term Unity</h2>
<p>Once we’ve made the changes detailed above, we’ll continue looking for ways we can open up Gradle ecosystem to even greater community contribution. We’re really excited about this, and we hope you will be too.</p>
<p>We’d love to hear your feedback—let us know what you think in the comments on this post or talk to us <a href="https://twitter.com/gradle">@Gradle</a> on Twitter. Your suggestions, questions, and of course, <a href="https://github.com/gradle/gradle/issues">issues</a> are most welcome.</p>
Introducing Composite Builds2016-10-12T00:00:00-04:00https://blog.gradle.org/introducing-composite-buildsStefan Oehme
<p>It’s not every day that we get to announce a feature that revolutionizes several software engineering workflows, but today is that day. <em><a href="https://docs.gradle.org/current/userguide/composite_builds.html">Composite builds</a>,</em> a new feature in Gradle 3.1, enables an entirely new dimension in project organization.</p>
<p>Composite builds are a way to <em>join multiple independent Gradle builds and build them together.</em> The brevity of that statement does not fully convey all of the new possibilities, so let me show you how this will make your life as a developer a lot easier.</p>
<h2 id="joining-projects">Joining projects</h2>
<p>Many organizations split their code base into several independent projects, each having a dedicated repository and release cycle. Integration between the projects is managed using binary dependencies, e.g. JAR files published to a binary repository like Artifactory. This approach has many advantages, but can be inefficient when trying to rapidly develop and test changes that affect two or more of these projects at once.</p>
<p>Imagine for a moment that you are fixing a bug in a Java library that your application depends on. Your workflow probably looks something like the following:</p>
<ol>
<li>Change the library</li>
<li>Publish the library to a local repository</li>
<li>Add the local repository to your application’s repositories</li>
<li>Change your application’s dependency to the new library version</li>
<li>Test your application</li>
<li>Repeat until the problem is fixed or you lose your mind</li>
</ol>
<p>With composite builds, you can short-circuit this workflow by <em>including the library’s build into your application’s build.</em> Gradle will then automatically replace the <em>binary</em> dependency on the library with a <em>project</em> dependency—meaning that changes you make to the library become available to the application instantaneously:</p>
<p style="text-align: center;"><script id="asciicast-9stxoe7vczhgkt0hhv6h6yk7e" src="https://asciinema.org/a/9stxoe7vczhgkt0hhv6h6yk7e.js" async="" type="text/javascript"></script></p>
<p>The same approach works for plugins that your project depends on. You can now include a locally checked-out version of a plugin into your project’s build and get into the same kind of tight development loop between them:</p>
<p style="text-align: center;"><script id="asciicast-cll4usx63ejj6aqocstebmp6q" src="https://asciinema.org/a/cll4usx63ejj6aqocstebmp6q.js" async="" type="text/javascript"></script></p>
<p>The new <code class="language-plaintext highlighter-rouge">includeBuild()</code> API in <code class="language-plaintext highlighter-rouge">settings.gradle</code> even lets you write a Gradle build that dynamically includes other builds if they are available on the local file system. You could then import this composite into your IDE and do cross-repository refactoring or debugging.</p>
<h2 id="splitting-monoliths">Splitting Monoliths</h2>
<p>Organizations that want to avoid the integration pains of multiple repositories tend to use a “monorepo”—a repository containing all projects, often including their dependencies and necessary tools. The upside is that all code is in one place and downstream breakages become visible immediately. But this convenience can come at the cost of productivity: a given developer will usually work only on a small part of a monorepo, but will still be forced to build all upstream projects, and that can mean a lot of waiting and wasted time. Likewise, importing large monorepo projects into an IDE often results in an unresponsive and overwhelming experience.</p>
<p>With composite builds, you can break your monorepo up into several independent builds within the same repository. Developers can work with the individual builds to get fast turnarounds or work with the whole composite when they want to ensure that everything still plays well together:</p>
<p style="text-align: center;"><script id="asciicast-6ze8hrt11ldevgonbue1sew3i" src="https://asciinema.org/a/6ze8hrt11ldevgonbue1sew3i.js" async="" type="text/javascript"></script></p>
<p>If you’re planning to move from a monolithic application to multiple independent ones, composite builds now offer a seamless migration strategy.</p>
<h2 id="this-is-just-the-beginning">This is just the beginning</h2>
<p>We plan to add a number of improvements to composite builds in upcoming releases:</p>
<ul>
<li>Targeting tasks in an included build from the command line</li>
<li>Richer dependency substitution API with support for custom publications</li>
<li>Executing included builds in parallel</li>
<li>Integration with Gradle’s <a href="https://docs.gradle.org/3.1/userguide/continuous_build.html">continuous build</a> capabilities</li>
<li>Out of the box support for composite builds in IntelliJ and Eclipse</li>
</ul>
<p>Of course, nothing is more important than feedback from real world usage. So please give composite builds a try in your own projects or have a look at the <a href="https://github.com/gradle/gradle/tree/v3.1.0/subprojects/docs/src/samples/compositeBuilds">samples</a>. Let us know about any problems, suggestions and cool things you built with it on the <a href="https://discuss.gradle.org">Gradle Forum</a>.</p>
Introducing Build Scans2016-10-06T00:00:00-04:00https://blog.gradle.org/introducing-build-scansChris Beams
<p>A few months ago at this year’s Gradle Summit conference, we announced a new part of the Gradle platform called <a href="https://www.youtube.com/watch?v=8IYA33e4_yk"><em>Gradle Cloud Services</em></a>. In this post, I want to introduce you to the first of these services—the <em>Gradle Build Scan Service</em>—and the <em>build scans</em> it makes possible.</p>
<h2 id="what-is-a-build-scan">What is a build scan?</h2>
<p>A build scan is a representation of data captured as you run your build. The <a href="https://plugins.gradle.org/plugin/com.gradle.build-scan">Build Scan Plugin</a> does the work of capturing the data and sending it to the Build Scan Service. The service then transforms the data into <em>information</em> you can use and share with others. Here’s a quick example of using a build scan to investigate a failure:</p>
<p><img src="https://blog.gradle.org/images/build-scan.gif" alt="Publishing and Viewing a Build Scan" /></p>
<!--break-->
<p>As you can see, the information that scans provide can be a big help when troubleshooting, collaborating on, or optimizing the performance of your builds. For example, with a build scan in the mix, it’s no longer necessary to copy and paste error messages or include all the details about your environment each time you want to ask a question on Stack Overflow or the Gradle Forum. Instead, just include a link to your latest build scan. It contains much, if not all, of the information the person answering your question might need to know. It’ll save you both time, and they’ll probably thank you for it.</p>
<h2 id="whos-using-them">Who’s using them?</h2>
<p>We’re excited that a number of prominent open source projects like Hibernate and JUnit 5 have <a href="https://github.com/hibernate/hibernate-orm/commit/e3bee9e">already</a> <a href="https://github.com/junit-team/junit5#gradle-build-scans">integrated</a> build scans into their workflow. You can take a look through sample scans from each of these projects at <a href="https://gradle.com/#explore">gradle.com/explore</a>.</p>
<p><a href="https://gradle.com/#explore"><img src="https://blog.gradle.org/images/explore-build-scans.jpg" alt="Open Source Gradle Build Scans" /></a></p>
<h2 id="put-build-scans-to-use-for-yourself">Put build scans to use for yourself</h2>
<p>If you’re new to build scans, now is a great time to start using them. We’re continually rolling out new features, and we’ll cover each of them in subsequent posts. In the meantime, you can learn how to enable build scans for your existing projects via our <a href="https://scans.gradle.com/get-started">getting started instructions</a>, or get up and running with a sample project by cloning our <a href="https://github.com/gradle/gradle-build-scan-quickstart">quick start repository</a> and following the steps in its README.</p>
<p>Happy scanning, and we look forward to your feedback!</p>
Hello, Again2016-09-26T00:00:00-04:00https://blog.gradle.org/hello-againChris Beams
<p>Welcome to the new Gradle blog. On behalf of the whole team, it’s my pleasure to write this first post and share a bit about what we’re up to.</p>
<p>Mainly, we’re putting together this new blog because we want to make sure users find out about the most important developments in Gradle-land. Our team has grown rapidly over the last couple years, and as a result we’ve shipped many new features and improvements. All too often, though, we find that would-be users never hear about them. This recent tweet provides a perfect example:</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">I read news, I try to keep up-to-date. So how did I miss that <a href="https://twitter.com/gradle">@gradle</a> can build <a href="https://twitter.com/playframework">@playframework</a>? Gotta look into this.</p>— Cristian Duicu (@cduicu) <a href="https://twitter.com/cduicu/status/779301788654723072">September 23, 2016</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Cristian’s question is a fair one. We first shipped Play support over a year ago; we mentioned it in our Gradle 2.6 <a href="https://discuss.gradle.org/t/gradle-2-6-released/11092">forum announcement</a> and <a href="https://docs.gradle.org/2.6/release-notes">release notes</a>, and we wrote <a href="https://docs.gradle.org/3.1/userguide/play_plugin.html">a chapter about it</a> in our user manual. Still, Cristian—and probably many others—missed it. How is that?</p>
<!--break-->
<p>The answer is pretty straightforward. Forum announcements and release notes are useful documents, but they get buried as new releases pile up. Reference documentation is important too, but our user manual has grown large, meaning that <em>Chapter 70: Building Play Applications</em> is easy to miss if you’re not already looking for it.</p>
<p>This situation isn’t surprising, nor is it unique to Gradle. As any project grows, it becomes a challenge to communicate about it effectively. In any case, we can no longer expect every current and future Gradle user to dig through past release notes or to read our user manual cover to cover simply to discover what Gradle can do. It’s on us to find better ways to get the word out.</p>
<p>And that’s where this new blog comes in. We’ll post here whenever there’s something new in Gradle that we don’t want you to miss. We’ll keep it focused on things we think are important and worth your time. We hope it’ll become a trusted resource—not only for you to stay up to date with Gradle, but also for us to get feedback through your comments.</p>
<p>A better blog isn’t a silver bullet, but we think it’s a great place to start. Indeed, it’s just the first step in a much larger effort to make working with and learning about Gradle as easy and enjoyable as possible. In the weeks to come you’ll see further improvements, including new guides and tutorials, a new gradle.org website, and a simpler process for filing bugs and feature requests.</p>
<p>Naturally, we’ll announce each of these changes here on the blog. To stay tuned, <a href="https://feed.gradle.org/blog.atom">subscribe to the blog’s Atom feed</a> or <a href="https://twitter.com/intent/follow?screen_name=Gradle">follow @Gradle on Twitter</a>, where we’ll link to new posts as we publish them.</p>