# Post mortem: Checkly security incident [updated] > This post mortem contains investigating notes of what seemed to be an Auth0 error & the strategy for future mitigation. Source: https://www.checklyhq.com/blog/post-mortem-security-incident/ --- [Blog](https://www.checklyhq.com/blog/) / [Security](https://www.checklyhq.com/blog/tag/security/) # Post mortem: Checkly security incident [updated] [Tim Nolet](https://www.checklyhq.com/blog/author/tim-nolet/) October 29, 2019 · Updated October 17, 2024 ## Update 31-10-2019 Today, a customer reported that on public dashboards, the contents of any browser check script are part of the API response that supplies the dashboard with data. Admittedly, this was by design not due to a bug or hack: In our thinking, browser check scripts should not contain sensitive data. Instead, secrets should be stored in the environment variables which are encrypted on disk and in flight. However, this design was too naïve and did not take into account that users might "hard code" sensitive data in their scripts. Also, the script data did not add anything to the dashboards and should have been scrubbed in the first place. For this reason, we consider this part of the security incident as in both cases unauthorized users could access this data. This issue was fixed today at 19:52 CET. ### Action If you showed browser checks on a public dashboard and you stored any confidential information directly in your browser check script, please take the following steps: - Rotate or change any passwords, block users or take any other measure to ensure the secrets used in your browser scripts are no longer valuable. - Remove the secrets in your script and stored them as global environment variables or environment variables directly in your browser check. Please see [the relevant docs on our docs site](https://www.checklyhq.com/docs/browser-checks/variables/) for help on how to achieve this. Similar to the parent incident, we have no reason to believe any check data was actually leaked. - No user data, emails, passwords or other Personally Identifiable Information was compromised. - No credit card data or other payment information was compromised. ### Future mitigation The mitigating steps to not have similar issues in the future are still relevant and listed below in the **What now?**paragraph. ## Original 29-10-2019 Today, from 16:17 till 18:21, users were able to see other users' disabled checks in their dashboard when setting a specific dashboard filter combination. When inspecting the JSON payload, they _could_ have read specific parts of check data. Specifically, if a browser check script contained any non-encrypted user credentials for test users, these might have been compromised. No user data, emails, passwords or other Personally Identifiable Information was compromised. No credit card data or other payment information was compromised. No encrypted variables, headers or other check fields were compromised. We have emailed all users that have been impacted purely as a precaution. We have no indication any data was actually leaked. ## Full timeline ### Notification At 17:30 CET I was made aware of security flaw in Checkly. This flaw allowed users to see disabled checks in their dashboard when configuring the dashboard filters to show **disabled** **checks** and **non**-**disabled** **checks**. This was clearly a priority 1 incident. ### Triage From **17:30**to roughly **17:55** I chatted on Intercom with the user reporting the issue to pin point the origin. At first I suspected some caching issue with our authentication provider Auth0, but I quickly discarded this as the behavior was 100% reproducible on a local dev machine. At **18:00** the issue became clear. It was a logic bug in the endpoint that servers the dashboard. Each query in fired on the database is accompanied by an `accountId`. This accountId determines which checks belong to which account. However, in one part of the query — where we determine the filter for disabled and other checks — a wrongly nested `OR` statement effectively negated the crucial `WHERE accountId` clause. ### Patch A patch was made, tested and rolled out. Our deployment log shows it went live at **18:21 CET.**Both the reporter and I confirmed the patch correctly addressed the issue. ### Impact assessment This issue has the following impact characteristics - It was live from 16:17 to 18:21 CET on 29-10-2020. - It only impacted one API endpoint that populated the dashboard. - It could only leak information on disabled checks, specifically the browser check scripts which contain free form data. Users _might_ enter test credentials there, although Checkly has many features to protect these credentials and store them **separately** and **encrypted**. - Only authenticated users could potentially see this data. - No account data was impacted. No username, email addresses, passwords, credit card data etc. Checkly does not store any of these types of data. We use 3rd party services like **Stripe** and **Auth0** for this. I have an exact overview of which checks for which account potentially could have been visible. I've emailed these users and advised them to take action. ### What now? The bug was pretty terrible because it was a logic bug. Not a hack or undocumented feature. This has an upside and a downside. **The upside:** The fix was trivial. Our authentication is not broken as this **The downside:** It shows we need to integrate authentication, accounts and how accounts interact with endpoints in a more robust way. For this reason, I'll take the following actions - Move any and all account "pinning" outside of the general application logic. This way any queries or other code cannot interfere with "the accounting" take place on another abstraction level. - Do a security assessment with a pro security consultant / ethical hacker. - Move as quickly as possible to a code review practice. - Keep lines open for security bugs using a bug bounty program. For any questions, please reach out to me on tim [at] checklyhq.com [Tim Nolet Chief Evangelist](https://www.checklyhq.com/blog/author/tim-nolet/) In This Article - Update 31-10-2019 - Original 29-10-2019 - Full timeline Share on social ## Related Articles [Post mortem: failing checks us-west-1 April 29, 2020](https://www.checklyhq.com/blog/post-mortem-failing-checks-us-west-1/)[Post mortem: outage browser check results & alerting May 19, 2020](https://www.checklyhq.com/blog/post-mortem-outage-browser-check-results-alerting/)[Post mortem: checks with async IIFE reporting success incorrectly June 15, 2020](https://www.checklyhq.com/blog/post-mortem-checks-with-async-iife-reporting-success-incorrectly/)[Changelog: Alerting, webhook, and Prometheus updates January 8, 2020](https://www.checklyhq.com/blog/changelog-alerting-webhook-and-prometheus-updates/)[Introducing Groups for your app monitoring to Checkly February 25, 2020](https://www.checklyhq.com/blog/introducing-groups/)