Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve debuggability #130

Open
lawrencegripper opened this issue Jul 4, 2018 · 2 comments
Open

Improve debuggability #130

lawrencegripper opened this issue Jul 4, 2018 · 2 comments
Labels
dispatcher Issues relating to the dispatcher enhancement New feature or request

Comments

@lawrencegripper
Copy link
Owner

Currently the dispatcher removes jobs (and hence their pods and the pods logs) when they have failed or completed.

This makes debugging a failed job very hard as the logs are logs. I

I propose moving to leave the jobs and then clean up any jobs that are older than 1 hour and have failed/completed.

Also the logging isn't great a lots of points and byte arrays are logged which aren't helpful, planning to clean this up too.

@lawrencegripper lawrencegripper added enhancement New feature or request dispatcher Issues relating to the dispatcher labels Jul 4, 2018
@lawrencegripper lawrencegripper self-assigned this Jul 4, 2018
lawrencegripper added a commit that referenced this issue Jul 4, 2018
@lawrencegripper lawrencegripper changed the title Improve dispatcher k8s provider debuggability Improve dispatcher and k8s provider debuggability Jul 4, 2018
@lawrencegripper lawrencegripper changed the title Improve dispatcher and k8s provider debuggability Improve debuggability Jul 6, 2018
@lawrencegripper
Copy link
Owner Author

I've added some additional debug output to the committer as I'm seeing an issue where events appear to be lost between modules.

@lawrencegripper
Copy link
Owner Author

I'd like to add an additional doctype to the work in #135 that an be used by the ion trace command I mentioned in #122. This would create a document storing the stdout and stderr from the module runs. I propose that these types have a ttl so they automatically get deleted by cosmos.

This would allow ion trace to show eventMeta, insights and failures for a given correlation id in one place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dispatcher Issues relating to the dispatcher enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant