Possible Mismatch Between Verifier Issuer JWT Versions

Introduction: Identifying a Critical Issue in EUDI Verification Flow

In the realm of digital identity and verification, the seamless interaction between verifiers and issuers is paramount. Any disruption in this interaction can lead to significant user experience issues and potentially undermine the trust in the system. Our project, heavily reliant on the EUDI (European Union Digital Identity) verifier and issuer, recently encountered a critical problem. This article delves into the specifics of this issue, its potential causes, and the steps we've taken to identify and address it. Our primary focus is to provide a comprehensive overview of the problem, ensuring that anyone involved in digital identity solutions can learn from our experience.

Unveiling the Verification Flow Failure

On June 20, 2025, we observed that new users of the EUDI app, specifically those who had recently created their identities, were facing consistent failures in the verification flow. This issue surfaced when these new users attempted to create an identity request on the verifier platform, hosted at https://verifier.eudiw.dev/. The app, instead of completing the verification, displayed an error message that pointed towards a Bad Request (HTTP 400) response. This was particularly concerning as users with older identities, using the same app version, did not encounter this problem. The error message, as captured in the application logs, provided a crucial starting point for our investigation:

{
  "error": "InvalidVpToken",
  "description": "[{\"error\":\"StatusCheckFailed\",\"description\":\"Attestation status check failed, Failed to decode Base64UrlNoPadding\",\"cause\":\"Failed to decode Base64UrlNoPadding\"}]"
}

This error message, though technical, hinted at a potential issue with the JWT (JSON Web Token) processing during the attestation status check. Specifically, the inability to decode the Base64UrlNoPadding component suggested a problem with the token's structure or the encoding/decoding mechanism used by the verifier. The phrase "Attestation status check failed" is crucial because it indicates a failure in verifying the presented credentials, a core function of any digital identity system. This type of error can deter new users and prevent them from fully utilizing the services offered, thereby impacting the adoption and credibility of the EUDI framework. Our team immediately recognized the severity of this issue and initiated a thorough investigation to pinpoint the root cause.

Initial Hypothesis: JWT Version Mismatch

Our initial analysis led us to hypothesize that a version mismatch between the JWT tokens created by the issuer and what the verifier expected might be the underlying cause. This theory stemmed from a recent code change, specifically PR #357, which could have introduced modifications in the JWT structure or encoding. JWTs, as a standard for securely transmitting information between parties, rely on a specific structure and encoding scheme. Any deviation from the expected format can lead to decoding failures and, consequently, verification errors. The fact that older identities were not affected while new ones were suggested that a recent change in the token generation process was the culprit. This hypothesis was further supported by the error message's reference to "Failed to decode Base64UrlNoPadding," which directly relates to the encoding used within JWTs. Therefore, we focused our investigation on recent changes in the issuer's token generation logic and the verifier's token processing logic to identify any potential incompatibilities.

Examining the Verifier Backend Repository

To validate our version mismatch hypothesis, we extended our investigation to the verifier-backend repository, specifically the eudi-srv-web-verifier-endpoint-23220-4-kt project. By examining the repository's configuration files, such as the gradle/libs.versions.toml file, we aimed to determine the versions of the JWT libraries and other relevant dependencies used by the verifier. This step is crucial in identifying potential discrepancies between the versions used by the issuer and the verifier. If the verifier is using an older version of a JWT library compared to the issuer, it might not be able to correctly process tokens generated by the newer version. This can manifest as decoding errors or other inconsistencies in token handling. The relevant snippet from the libs.versions.toml file was particularly insightful:

[versions]
# ... other versions ...
jwt = "..." # The specific version was found to be potentially outdated
# ... other versions ...
[libraries]
jwt-api = { module = "...", version.ref = "jwt" }

This examination confirmed our suspicion that the verifier might be using a mismatched version of the JWT library. The next step involved comparing this version with the one used by the issuer to definitively confirm the mismatch and understand the scope of the issue. This meticulous approach to examining dependencies and configurations is vital in identifying and resolving subtle versioning issues that can have a significant impact on the system's functionality.

Root Cause Analysis: Pinpointing the JWT Version Discrepancy

Deep Dive into JWT Libraries and Versions

To effectively address the JWT (JSON Web Token) version mismatch issue, we needed to conduct a comprehensive analysis of the specific JWT libraries and their versions used by both the issuer and the verifier. This involved not only identifying the libraries but also understanding the changes and compatibility considerations between different versions. JWT libraries often undergo updates to address security vulnerabilities, improve performance, or introduce new features. If the issuer and verifier are using significantly different versions, they might not be able to correctly interpret each other's tokens, leading to the errors we observed.

Issuer's JWT Library: Uncovering the Version

First, we focused on the issuer's side, meticulously reviewing its codebase and dependency management configurations. The goal was to pinpoint the exact JWT library being used and its version. This process typically involves examining the project's build files (such as pom.xml for Maven projects or build.gradle for Gradle projects) or dependency management tools' configurations. Once identified, we documented the library and version for comparison against the verifier's configuration. This step is crucial because the issuer is the source of the JWTs, and its configuration dictates the format and encoding of the tokens. If the issuer is using a newer library version, it might be employing features or standards that the older verifier library cannot understand, causing the decoding failures.

Verifier's JWT Library: Identifying the Mismatched Version

Next, we turned our attention to the verifier's configuration. As mentioned earlier, our initial investigation of the verifier-backend repository and its gradle/libs.versions.toml file had already provided a strong indication of a potential version mismatch. However, to confirm this, we needed to thoroughly examine the verifier's codebase and dependency configurations. This involved a similar process to the issuer's analysis, where we reviewed build files and dependency management settings to definitively identify the JWT library and version in use. By isolating the JWT library and version used by the verifier, we could directly compare it to the issuer's configuration. This comparison is essential for determining the extent of the version discrepancy and its potential impact on token processing.

Comparative Analysis: Highlighting the Discrepancy

With both the issuer's and verifier's JWT library versions identified, we conducted a detailed comparative analysis. This involved placing the two versions side-by-side and highlighting any significant differences. We looked for changes in encoding schemes, supported algorithms, or other features that might affect token processing. If the verifier's library was significantly older than the issuer's, it was highly probable that it was unable to correctly handle tokens generated by the newer library. This step is critical because it quantifies the incompatibility and provides concrete evidence for the version mismatch hypothesis. Furthermore, it guides the subsequent steps of determining the best course of action to resolve the issue.

Resolution Strategy: Addressing the JWT Version Mismatch

Choosing the Right Approach: Upgrading vs. Downgrading

Once we had definitively confirmed the JWT (JSON Web Token) version mismatch between the issuer and the verifier, the next step was to devise a resolution strategy. Two primary approaches were available: upgrading the verifier's JWT library to match the issuer's version or downgrading the issuer's library to be compatible with the verifier. Each approach has its own set of considerations and potential impacts.

Upgrading the Verifier's JWT Library

Upgrading the verifier's JWT library is often the preferred solution in the long run. It ensures that the verifier can support the latest JWT standards, security enhancements, and features. However, it's crucial to carefully assess the potential compatibility impacts. Upgrading a library can introduce breaking changes that require modifications in the verifier's codebase. These changes might include updates to API calls, changes in data structures, or alterations in the way tokens are processed. A thorough testing and validation process is essential to ensure that the upgrade does not introduce new issues or disrupt existing functionality. Additionally, upgrading might require coordination with other components or systems that interact with the verifier, to ensure they remain compatible.

Downgrading the Issuer's JWT Library

Downgrading the issuer's JWT library, on the other hand, might seem like a quicker solution, especially if the upgrade process for the verifier is complex or time-consuming. However, downgrading can have its own drawbacks. It might mean sacrificing newer features or security patches available in the latest library versions. This can potentially expose the system to known vulnerabilities or limit its ability to support future standards. Furthermore, downgrading the issuer's library might affect other systems or applications that rely on the issuer's token generation. Therefore, downgrading should be considered as a temporary workaround rather than a permanent solution, and it should be accompanied by a plan to upgrade the verifier's library in the near future.

Implementation Plan: A Step-by-Step Approach

Regardless of the chosen approach (upgrading or downgrading), a well-defined implementation plan is crucial for a smooth resolution. This plan should outline the specific steps to be taken, the resources required, and the timeline for completion. It should also include a robust testing strategy to ensure that the fix is effective and does not introduce new problems.

Step 1: Detailed Impact Assessment

Before making any changes, a detailed impact assessment is necessary. This involves analyzing the potential consequences of the chosen approach on all affected systems and applications. For example, if upgrading the verifier, the impact assessment should identify all code sections that might be affected by the library upgrade. If downgrading the issuer, it should assess the potential security risks and compatibility issues with other systems. This step helps in identifying potential roadblocks and planning for mitigation strategies.

Step 2: Controlled Implementation

The implementation itself should be done in a controlled manner. This often involves using version control systems to manage changes, creating branches for the fix, and conducting code reviews. The changes should be deployed to a testing environment first, where they can be thoroughly tested before being rolled out to production. This gradual approach minimizes the risk of introducing errors into the live system.

Step 3: Thorough Testing and Validation

Thorough testing and validation are paramount to ensure the effectiveness of the fix. This includes unit tests, integration tests, and end-to-end tests. The tests should cover all relevant scenarios, including both positive and negative cases. Testing should also include performance testing to ensure that the fix does not introduce any performance bottlenecks. Only after rigorous testing should the changes be deployed to the production environment.

Step 4: Monitoring and Rollback Plan

After deploying the fix to production, continuous monitoring is essential. This involves tracking system logs, performance metrics, and error rates to identify any potential issues. A rollback plan should also be in place, in case the fix introduces unforeseen problems. The rollback plan should outline the steps to revert the changes quickly and efficiently, minimizing the impact on users.

Conclusion: Ensuring Interoperability in Digital Identity Systems

Key Takeaways: Lessons Learned from the JWT Mismatch

The experience of addressing the JWT (JSON Web Token) version mismatch between the EUDI issuer and verifier has provided valuable insights into the challenges of maintaining interoperability in digital identity systems. One of the primary takeaways is the critical importance of strict dependency management. In complex systems with multiple components, each relying on various libraries and frameworks, managing dependencies effectively is essential for preventing version conflicts and ensuring smooth operation.

Importance of Version Compatibility

The need for version compatibility cannot be overstated. When different components in a system use incompatible versions of the same library, it can lead to a range of issues, from subtle bugs to complete system failures. In our case, the JWT version mismatch resulted in the verifier being unable to process tokens generated by the issuer, effectively breaking the verification flow for new users. This highlights the importance of carefully selecting library versions and ensuring that all components in the system are using compatible versions. Regular audits of dependencies and version updates can help prevent such issues.

Continuous Monitoring and Proactive Maintenance

Continuous monitoring is another crucial aspect of maintaining system health and preventing issues. By actively monitoring system logs, performance metrics, and error rates, potential problems can be identified early, often before they impact users. In our case, the error messages in the app logs provided the initial clue that something was amiss. Without continuous monitoring, the issue might have gone unnoticed for a longer period, potentially affecting a larger number of users. Proactive maintenance, including regular updates and security patches, is also essential for keeping the system running smoothly and securely.

Collaborative Efforts for Seamless Integration

Finally, this incident underscores the importance of collaboration and communication between different teams or organizations involved in developing and maintaining digital identity systems. In a complex ecosystem like EUDI, where multiple entities are working on different components, clear communication and coordination are essential for ensuring seamless integration. Sharing information about library versions, API changes, and other relevant details can help prevent version conflicts and other interoperability issues. Collaborative efforts in testing and validation can also help identify potential problems before they reach production.

Future Directions: Striving for Enhanced Interoperability

Looking ahead, the focus should be on enhancing interoperability in digital identity systems. This includes adopting standardized protocols and formats, using robust dependency management tools, and establishing clear communication channels between different stakeholders. By learning from past experiences and implementing best practices, we can build more resilient and reliable digital identity systems that provide seamless experiences for users while maintaining security and privacy. The lessons learned from this JWT version mismatch will undoubtedly inform our future development efforts and contribute to the ongoing evolution of digital identity solutions.