Please deduplicate results

[duck-rh]

Is your feature request related to a problem? Please describe.
Warnings and suggestions are great but appear multiple times with the same exact message.

Some suggestions like NETW-3200 appear multiple times, one for each protocol, that makes sense, but some like KRNL-5830 or HTTP-6660 are identical and appear multiple times.

Also note that the way detection works makes all these duplicate messages to appear in different order and it's not just a matter of skipping over identical lines.

Describe the solution you'd like
It would be easier if a specific code appeared only once.

I understand that for NETW-3200 it would be problematic and I think that would need to be split into one code per protocol.
But that's slightly out of scope of this feature request although tightly connected.

If the code and message are the same I would prefer that it appears only once.

Required changes
Ideally that a code is unique and not shared by multiple messages.

That a message only appears once in the report.

Additional context

Example report:

  -[ Lynis 3.1.2 Results ]-

  Warnings (1):
  ----------------------------
  ! Reboot of system is most likely needed [KRNL-5830] 
    - Solution : reboot
      https://cisofy.com/lynis/controls/KRNL-5830/

  ! Reboot of system is most likely needed [KRNL-5830] 
    - Solution : reboot
      https://cisofy.com/lynis/controls/KRNL-5830/

  Suggestions (24):
  ----------------------------
  * Copy /etc/fail2ban/jail.conf to jail.local to prevent it being changed by updates. [DEB-0880] 
      https://cisofy.com/lynis/controls/DEB-0880/

  * Consider hardening system services [BOOT-5264] 
    - Details  : Run '/usr/bin/systemd-analyze security SERVICE' for each service
      https://cisofy.com/lynis/controls/BOOT-5264/

  * Copy /etc/fail2ban/jail.conf to jail.local to prevent it being changed by updates. [DEB-0880] 
      https://cisofy.com/lynis/controls/DEB-0880/

  * If not required, consider explicit disabling of core dump in /etc/security/limits.conf file [KRNL-5820] 
      https://cisofy.com/lynis/controls/KRNL-5820/

  * Check PAM configuration, add rounds if applicable and expire passwords to encrypt with new values [AUTH-9229] 
      https://cisofy.com/lynis/controls/AUTH-9229/

  * Configure password hashing rounds in /etc/login.defs [AUTH-9230] 
      https://cisofy.com/lynis/controls/AUTH-9230/

  * When possible set expire dates for all password protected accounts [AUTH-9282] 
      https://cisofy.com/lynis/controls/AUTH-9282/

  * Look at the locked accounts and consider removing them [AUTH-9284] 
      https://cisofy.com/lynis/controls/AUTH-9284/

  * Consider hardening system services [BOOT-5264] 
    - Details  : Run '/usr/bin/systemd-analyze security SERVICE' for each service
      https://cisofy.com/lynis/controls/BOOT-5264/

  * Check 1022 files in /tmp which are older than 90 days [FILE-6354] 
      https://cisofy.com/lynis/controls/FILE-6354/

  * If not required, consider explicit disabling of core dump in /etc/security/limits.conf file [KRNL-5820] 
      https://cisofy.com/lynis/controls/KRNL-5820/

  * Check PAM configuration, add rounds if applicable and expire passwords to encrypt with new values [AUTH-9229] 
      https://cisofy.com/lynis/controls/AUTH-9229/

  * Configure password hashing rounds in /etc/login.defs [AUTH-9230] 
      https://cisofy.com/lynis/controls/AUTH-9230/

  * When possible set expire dates for all password protected accounts [AUTH-9282] 
      https://cisofy.com/lynis/controls/AUTH-9282/

  * Look at the locked accounts and consider removing them [AUTH-9284] 
      https://cisofy.com/lynis/controls/AUTH-9284/

  * Check 1022 files in /tmp which are older than 90 days [FILE-6354] 
      https://cisofy.com/lynis/controls/FILE-6354/

  * Purge old/removed packages (35 found) with aptitude purge or dpkg --purge command. This will cleanup old configuration files, cron jobs and startup scripts. [PKGS-7346] 
      https://cisofy.com/lynis/controls/PKGS-7346/

  * Remove any unneeded kernel packages [PKGS-7410] 
    - Details  : 37 kernels
    - Solution : validate dpkg -l output and perform cleanup with apt autoremove
      https://cisofy.com/lynis/controls/PKGS-7410/

  * Purge old/removed packages (35 found) with aptitude purge or dpkg --purge command. This will cleanup old configuration files, cron jobs and startup scripts. [PKGS-7346] 
      https://cisofy.com/lynis/controls/PKGS-7346/

  * Determine if protocol 'sctp' is really needed on this system [NETW-3200] 
      https://cisofy.com/lynis/controls/NETW-3200/

  * Determine if protocol 'rds' is really needed on this system [NETW-3200] 
      https://cisofy.com/lynis/controls/NETW-3200/

  * Determine if protocol 'tipc' is really needed on this system [NETW-3200] 
      https://cisofy.com/lynis/controls/NETW-3200/

  * Install Apache mod_evasive to guard webserver against DoS/brute force attempts [HTTP-6640] 
      https://cisofy.com/lynis/controls/HTTP-6640/

  * Install Apache modsecurity to guard webserver against web application attacks [HTTP-6643] 
      https://cisofy.com/lynis/controls/HTTP-6643/

  * Remove any unneeded kernel packages [PKGS-7410] 
    - Details  : 37 kernels
    - Solution : validate dpkg -l output and perform cleanup with apt autoremove
      https://cisofy.com/lynis/controls/PKGS-7410/

  * Consider setting 'TraceEnable Off' in /etc/apache2/conf-enabled/common.conf [HTTP-6660] 
    - Details  : Set TraceEnable to 'On' or 'extended' for testing and diagnostic purposes only.
      https://cisofy.com/lynis/controls/HTTP-6660/

  * Consider setting 'TraceEnable Off' in /etc/apache2/conf-available/security.conf [HTTP-6660] 
    - Details  : Set TraceEnable to 'On' or 'extended' for testing and diagnostic purposes only.
      https://cisofy.com/lynis/controls/HTTP-6660/

  * Change the allow_url_fopen line to: allow_url_fopen = Off, to disable downloads via PHP [PHP-2376] 
      https://cisofy.com/lynis/controls/PHP-2376/

  * Determine if protocol 'sctp' is really needed on this system [NETW-3200] 
      https://cisofy.com/lynis/controls/NETW-3200/

  * Determine if protocol 'rds' is really needed on this system [NETW-3200] 
      https://cisofy.com/lynis/controls/NETW-3200/

  * Determine if protocol 'tipc' is really needed on this system [NETW-3200] 
      https://cisofy.com/lynis/controls/NETW-3200/

  * Enable logging to an external logging host for archiving purposes and additional protection [LOGG-2154] 
      https://cisofy.com/lynis/controls/LOGG-2154/

  * Install Apache mod_evasive to guard webserver against DoS/brute force attempts [HTTP-6640] 
      https://cisofy.com/lynis/controls/HTTP-6640/

  * Install Apache modsecurity to guard webserver against web application attacks [HTTP-6643] 
      https://cisofy.com/lynis/controls/HTTP-6643/

  * Audit daemon is enabled with an empty ruleset. Disable the daemon or define rules [ACCT-9630] 
      https://cisofy.com/lynis/controls/ACCT-9630/

  * Consider setting 'TraceEnable Off' in /etc/apache2/conf-enabled/common.conf [HTTP-6660] 
    - Details  : Set TraceEnable to 'On' or 'extended' for testing and diagnostic purposes only.
      https://cisofy.com/lynis/controls/HTTP-6660/

  * Consider setting 'TraceEnable Off' in /etc/apache2/conf-available/security.conf [HTTP-6660] 
    - Details  : Set TraceEnable to 'On' or 'extended' for testing and diagnostic purposes only.
      https://cisofy.com/lynis/controls/HTTP-6660/

  * Check available certificates for expiration [CRYP-7902] 
      https://cisofy.com/lynis/controls/CRYP-7902/

  * Change the allow_url_fopen line to: allow_url_fopen = Off, to disable downloads via PHP [PHP-2376] 
      https://cisofy.com/lynis/controls/PHP-2376/

  * Install a file integrity tool to monitor changes to critical and sensitive files [FINT-4350] 
      https://cisofy.com/lynis/controls/FINT-4350/

  * Enable logging to an external logging host for archiving purposes and additional protection [LOGG-2154] 
      https://cisofy.com/lynis/controls/LOGG-2154/

  * Consider restricting file permissions [FILE-7524] 
    - Details  : See screen output or log file
    - Solution : Use chmod to change file permissions
      https://cisofy.com/lynis/controls/FILE-7524/

  * Audit daemon is enabled with an empty ruleset. Disable the daemon or define rules [ACCT-9630] 
      https://cisofy.com/lynis/controls/ACCT-9630/

  * Check available certificates for expiration [CRYP-7902] 
      https://cisofy.com/lynis/controls/CRYP-7902/

  * One or more sysctl values differ from the scan profile and could be tweaked [KRNL-6000] 
    - Solution : Change sysctl value or disable test (skip-test=KRNL-6000:<sysctl-key>)
      https://cisofy.com/lynis/controls/KRNL-6000/

  * Install a file integrity tool to monitor changes to critical and sensitive files [FINT-4350] 
      https://cisofy.com/lynis/controls/FINT-4350/

  * Consider restricting file permissions [FILE-7524] 
    - Details  : See screen output or log file
    - Solution : Use chmod to change file permissions
      https://cisofy.com/lynis/controls/FILE-7524/

  * One or more sysctl values differ from the scan profile and could be tweaked [KRNL-6000] 
    - Solution : Change sysctl value or disable test (skip-test=KRNL-6000:<sysctl-key>)
      https://cisofy.com/lynis/controls/KRNL-6000/

[wileyhy]

I agree, deduplication would save users' some time. A lot of people might value something like that. Meanwhile, here's a workaround for you. :-) Cheers, Wiley ~ $ head -v lynis.log ==> lynis.log <== -[ Lynis 3.1.2 Results ]- Warnings (1):

…

---------------------------- ! Reboot of system is most likely needed [KRNL-5830] - Solution : reboot https://cisofy.com/lynis/controls/KRNL-5830/ ! Reboot of system is most likely needed [KRNL-5830] - Solution : reboot ~ $ wc lynis.log 174 843 8295 lynis.log ~ $ sort -u lynis.log | wc 60 408 3905 ~ $ awk '!a[$0]++' lynis.log | wc 60 408 3905 ~ $ awk '!a[$0]++' lynis.log | head -v ==> standard input <== -[ Lynis 3.1.2 Results ]- Warnings (1):

---------------------------- ! Reboot of system is most likely needed [KRNL-5830] - Solution : reboot https://cisofy.com/lynis/controls/KRNL-5830/ Suggestions (24): * Copy /etc/fail2ban/jail.conf to jail.local to prevent it being changed by updates. [DEB-0880] https://cisofy.com/lynis/controls/DEB-0880/

On Tue, Nov 12, 2024, 17:28 Marc Dequènes (Duck) ***@***.***> wrote: *Is your feature request related to a problem? Please describe.* Warnings and suggestions are great but appear multiple times with the same exact message. Some suggestions like NETW-3200 appear multiple times, one for each protocol, that makes sense, but some like KRNL-5830 or HTTP-6660 are identical and appear multiple times. Also note that the way detection works makes all these duplicate messages to appear in different order and it's not just a matter of skipping over identical lines. *Describe the solution you'd like* It would be easier if a specific code appeared only once. I understand that for NETW-3200 it would be problematic and I think that would need to be split into one code per protocol. But that's slightly out of scope of this feature request although tightly connected. If the code and message are the same I would prefer that it appears only once. *Required changes* Ideally that a code is unique and not shared by multiple messages. That a message only appears once in the report. *Additional context* Example report: -[ Lynis 3.1.2 Results ]- Warnings (1): ---------------------------- ! Reboot of system is most likely needed [KRNL-5830] - Solution : reboot https://cisofy.com/lynis/controls/KRNL-5830/ ! Reboot of system is most likely needed [KRNL-5830] - Solution : reboot https://cisofy.com/lynis/controls/KRNL-5830/ Suggestions (24): ---------------------------- * Copy /etc/fail2ban/jail.conf to jail.local to prevent it being changed by updates. [DEB-0880] https://cisofy.com/lynis/controls/DEB-0880/ * Consider hardening system services [BOOT-5264] - Details : Run '/usr/bin/systemd-analyze security SERVICE' for each service https://cisofy.com/lynis/controls/BOOT-5264/ * Copy /etc/fail2ban/jail.conf to jail.local to prevent it being changed by updates. [DEB-0880] https://cisofy.com/lynis/controls/DEB-0880/ * If not required, consider explicit disabling of core dump in /etc/security/limits.conf file [KRNL-5820] https://cisofy.com/lynis/controls/KRNL-5820/ * Check PAM configuration, add rounds if applicable and expire passwords to encrypt with new values [AUTH-9229] https://cisofy.com/lynis/controls/AUTH-9229/ * Configure password hashing rounds in /etc/login.defs [AUTH-9230] https://cisofy.com/lynis/controls/AUTH-9230/ * When possible set expire dates for all password protected accounts [AUTH-9282] https://cisofy.com/lynis/controls/AUTH-9282/ * Look at the locked accounts and consider removing them [AUTH-9284] https://cisofy.com/lynis/controls/AUTH-9284/ * Consider hardening system services [BOOT-5264] - Details : Run '/usr/bin/systemd-analyze security SERVICE' for each service https://cisofy.com/lynis/controls/BOOT-5264/ * Check 1022 files in /tmp which are older than 90 days [FILE-6354] https://cisofy.com/lynis/controls/FILE-6354/ * If not required, consider explicit disabling of core dump in /etc/security/limits.conf file [KRNL-5820] https://cisofy.com/lynis/controls/KRNL-5820/ * Check PAM configuration, add rounds if applicable and expire passwords to encrypt with new values [AUTH-9229] https://cisofy.com/lynis/controls/AUTH-9229/ * Configure password hashing rounds in /etc/login.defs [AUTH-9230] https://cisofy.com/lynis/controls/AUTH-9230/ * When possible set expire dates for all password protected accounts [AUTH-9282] https://cisofy.com/lynis/controls/AUTH-9282/ * Look at the locked accounts and consider removing them [AUTH-9284] https://cisofy.com/lynis/controls/AUTH-9284/ * Check 1022 files in /tmp which are older than 90 days [FILE-6354] https://cisofy.com/lynis/controls/FILE-6354/ * Purge old/removed packages (35 found) with aptitude purge or dpkg --purge command. This will cleanup old configuration files, cron jobs and startup scripts. [PKGS-7346] https://cisofy.com/lynis/controls/PKGS-7346/ * Remove any unneeded kernel packages [PKGS-7410] - Details : 37 kernels - Solution : validate dpkg -l output and perform cleanup with apt autoremove https://cisofy.com/lynis/controls/PKGS-7410/ * Purge old/removed packages (35 found) with aptitude purge or dpkg --purge command. This will cleanup old configuration files, cron jobs and startup scripts. [PKGS-7346] https://cisofy.com/lynis/controls/PKGS-7346/ * Determine if protocol 'sctp' is really needed on this system [NETW-3200] https://cisofy.com/lynis/controls/NETW-3200/ * Determine if protocol 'rds' is really needed on this system [NETW-3200] https://cisofy.com/lynis/controls/NETW-3200/ * Determine if protocol 'tipc' is really needed on this system [NETW-3200] https://cisofy.com/lynis/controls/NETW-3200/ * Install Apache mod_evasive to guard webserver against DoS/brute force attempts [HTTP-6640] https://cisofy.com/lynis/controls/HTTP-6640/ * Install Apache modsecurity to guard webserver against web application attacks [HTTP-6643] https://cisofy.com/lynis/controls/HTTP-6643/ * Remove any unneeded kernel packages [PKGS-7410] - Details : 37 kernels - Solution : validate dpkg -l output and perform cleanup with apt autoremove https://cisofy.com/lynis/controls/PKGS-7410/ * Consider setting 'TraceEnable Off' in /etc/apache2/conf-enabled/common.conf [HTTP-6660] - Details : Set TraceEnable to 'On' or 'extended' for testing and diagnostic purposes only. https://cisofy.com/lynis/controls/HTTP-6660/ * Consider setting 'TraceEnable Off' in /etc/apache2/conf-available/security.conf [HTTP-6660] - Details : Set TraceEnable to 'On' or 'extended' for testing and diagnostic purposes only. https://cisofy.com/lynis/controls/HTTP-6660/ * Change the allow_url_fopen line to: allow_url_fopen = Off, to disable downloads via PHP [PHP-2376] https://cisofy.com/lynis/controls/PHP-2376/ * Determine if protocol 'sctp' is really needed on this system [NETW-3200] https://cisofy.com/lynis/controls/NETW-3200/ * Determine if protocol 'rds' is really needed on this system [NETW-3200] https://cisofy.com/lynis/controls/NETW-3200/ * Determine if protocol 'tipc' is really needed on this system [NETW-3200] https://cisofy.com/lynis/controls/NETW-3200/ * Enable logging to an external logging host for archiving purposes and additional protection [LOGG-2154] https://cisofy.com/lynis/controls/LOGG-2154/ * Install Apache mod_evasive to guard webserver against DoS/brute force attempts [HTTP-6640] https://cisofy.com/lynis/controls/HTTP-6640/ * Install Apache modsecurity to guard webserver against web application attacks [HTTP-6643] https://cisofy.com/lynis/controls/HTTP-6643/ * Audit daemon is enabled with an empty ruleset. Disable the daemon or define rules [ACCT-9630] https://cisofy.com/lynis/controls/ACCT-9630/ * Consider setting 'TraceEnable Off' in /etc/apache2/conf-enabled/common.conf [HTTP-6660] - Details : Set TraceEnable to 'On' or 'extended' for testing and diagnostic purposes only. https://cisofy.com/lynis/controls/HTTP-6660/ * Consider setting 'TraceEnable Off' in /etc/apache2/conf-available/security.conf [HTTP-6660] - Details : Set TraceEnable to 'On' or 'extended' for testing and diagnostic purposes only. https://cisofy.com/lynis/controls/HTTP-6660/ * Check available certificates for expiration [CRYP-7902] https://cisofy.com/lynis/controls/CRYP-7902/ * Change the allow_url_fopen line to: allow_url_fopen = Off, to disable downloads via PHP [PHP-2376] https://cisofy.com/lynis/controls/PHP-2376/ * Install a file integrity tool to monitor changes to critical and sensitive files [FINT-4350] https://cisofy.com/lynis/controls/FINT-4350/ * Enable logging to an external logging host for archiving purposes and additional protection [LOGG-2154] https://cisofy.com/lynis/controls/LOGG-2154/ * Consider restricting file permissions [FILE-7524] - Details : See screen output or log file - Solution : Use chmod to change file permissions https://cisofy.com/lynis/controls/FILE-7524/ * Audit daemon is enabled with an empty ruleset. Disable the daemon or define rules [ACCT-9630] https://cisofy.com/lynis/controls/ACCT-9630/ * Check available certificates for expiration [CRYP-7902] https://cisofy.com/lynis/controls/CRYP-7902/ * One or more sysctl values differ from the scan profile and could be tweaked [KRNL-6000] - Solution : Change sysctl value or disable test (skip-test=KRNL-6000:<sysctl-key>) https://cisofy.com/lynis/controls/KRNL-6000/ * Install a file integrity tool to monitor changes to critical and sensitive files [FINT-4350] https://cisofy.com/lynis/controls/FINT-4350/ * Consider restricting file permissions [FILE-7524] - Details : See screen output or log file - Solution : Use chmod to change file permissions https://cisofy.com/lynis/controls/FILE-7524/ * One or more sysctl values differ from the scan profile and could be tweaked [KRNL-6000] - Solution : Change sysctl value or disable test (skip-test=KRNL-6000:<sysctl-key>) https://cisofy.com/lynis/controls/KRNL-6000/ — Reply to this email directly, view it on GitHub <#1574>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUF2F244XNXQEYEOCKYFDBT2AKTKRAVCNFSM6AAAAABRVKK2T2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGY2TGOBTGA3DEMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mboelen]

Agree, duplicates should not be there. But... when I look at HTTP-6660, I don't see duplicates in the suggestion itself. Same for NETW-3200.

For example the two lines shown for HTTP-6660:

* Consider setting 'TraceEnable Off' in /etc/apache2/conf-enabled/common.conf [HTTP-6660] 
* Consider setting 'TraceEnable Off' in /etc/apache2/conf-available/security.conf [HTTP-6660]

So those are different, even though the suggestion is initiated both by the same test.

But you are right about KRNL-5830. So I suggest that we look into that test specifically, as the others have unique suggestion messages. Can you create a new issue for that one, including the relevant log section?