Bug #50012

Statserv: Bugs and improvements

Added by raztoki over 6 years ago. Updated over 4 years ago.

Status:AssignedStart date:08/25/2014
Priority:NormalDue date:
Assignee:coalado% Done:


Target version:030 - Version > 2.xxx


first thing that needs fixing,
after commits to given class' statserv drops existing issue tickets (typically plugin faults with subject title containing checksum md5 hash) and creates a new ticket with exact same logs and new stat graphic. This behaviour is incorrect, and should be address ASAP. It's burning through 1000s of issue tickets (for no reason, other than the stat graphic is new of revision but this doesn't reflect plotting with new revision data [1]). The worst thing is if you watch a ticket because your interested and expect to receive feedback, the ticket closes and you have to then find the new one...

[1] if it is for that reason, overlap colours reflecting revisions in single graphic. with the ability to turn off (checkbox) different revisions stats (No doubt would require different stat plotter), OR and or drop off stats from previous revisions at the point of of new revision spike. New stats should be of new revision reports.

second thing,
design on reporting issues should always analyse revision of classes (and abstracts, for example changes in K2SApi and not keep2share) to determine change. On new revision we should STOP accepting stats from old revisions. They should be stored and used as reference only, but more importantly the stat backend should give feedback to the client to update. This does two things, it should prevent the stat engine in JD reporting issues in outdated revisions, until it updates. This reduces traffic to the server, and database. It also ensures that all stats going forward are of the correct revision (or near abouts).

third thing,
do not delete/purge statserv tickets from the ticket system. I've assigned ticket and left comments on many tickets and next day they all disappear. This leaves little room for feedback or historical account. This is common issue related to 'first thing')

fourth thing,
stats plotter in the svn tickets need the time vertical (make the image vertically bigger to allow for this), the date/time references over lap and you can't read it. Normally a problem with newer graphics with small time interval. Would be nice if the time axis has a legend indicating what each block represents (specially when it's not readable).

fifth thing
not sure if its already done, I assume it has been.. But need make sure time is correctly offset in reports, if reports and logs are posted 10 hours earlier! It should plot stats 10 hours in the past, even if the issue ticket was just created (time reference should not be of when analysis has been carried out or ticket is created.)

sixth thing
make sure that one user can not create a issue ticket report. I've seen client submit multiple logs and suspect create all the Count values on plotter to a single person. This should not happen, a single JD install should only reflect a single count value in the database && statserv ticket (issue id md5checksum?) once per day. This is regardless of how many times the issue happens locally, or how many times the user restarts there client (should be based off JD installation UID). The threshold on reporting issues should be reflected based on popularity of site, as in if the site has less popularity(over all usage), it should take less reports about a given issue to create a statserv report/issue ticket. The more popular, the higher the required threshold! One person should not be able to influence statserve to report an issue. The reference I give was caused by a single user using a proxy (ip was in both logs reports, same country of origin and computer name etc).. We need to filter out these type of reports as they are bound ground noise. Ticket in question http://svn.jdownloader.org/issues/49793#note-1 lets hope that it doesn't get deleted.

seventh thing
Multihoster && when using real hoster plugin class for requestFileInformation && throwing plugin defect exception we need show ticket name as hoster and not the mulihoster. We have hundreds of multihoster tickets because of requestFileInformation in real hoster plugin! This is also reflecting badly for requestFileInformation in real hoster plugin reports (db/stats/svn issue tickets)! FIX!

eight thing
How should we treat proxies and plugin exceptions, for example this user has localhost loopback proxy

------------------------Thread: 669:fileom.com-----------------------
--ID:669TS:1408361780500-18/08/14 13.36.20 -  [jd.controlling.downloadcontroller.SingleDownloadController(run)] -> Start Download of http://fileom.com/removed
--ID:669TS:1408361780500-18/08/14 13.36.20 -  [jd.controlling.downloadcontroller.SingleDownloadController(download)] -> DownloadCandidate: DownloadCandidate:m4l3f1c3nt.dr.md.part2.rar|Host fileom.com|Account:Plugin:fileom.com|Version:26481|Type:NONE|Account:null|Proxy: (Http Proxy)
--ID:669TS:1408361781503-18/08/14 13.36.21 -  [jd.http.Browser(openRequestConnection)] -> 
----------------Request Information-------------
URL: http://fileom.com/removed
Host: fileom.com
Connection-Timeout: 20000ms
Read-Timeout: 60000ms
0:/|Connection refused: connect
-------------Not Connected Yet!-----------------

----------------Response Information------------
-------------Not Connected Yet!------------------

--ID:669TS:1408361781503-18/08/14 13.36.21 -  [] -> org.appwork.utils.net.httpconnection.ProxyConnectException: java.net.ConnectException: Connection refused: connect
    at org.appwork.utils.net.httpconnection.HTTPProxyHTTPConnectionImpl.connect(HTTPProxyHTTPConnectionImpl.java:76)
    at org.appwork.utils.net.httpconnection.HTTPConnectionImpl.finalizeConnect(HTTPConnectionImpl.java:641)
    at jd.http.Request.connect(Request.java:242)
    at jd.http.Browser.openRequestConnection(Browser.java:1286)
    at jd.http.Browser.openRequestConnection(Browser.java:1262)
    at jd.http.Browser.getPage(Browser.java:1003)
    at jd.plugins.hoster.FileOmCom.getPage(FileOmCom.java:1249)
    at jd.plugins.hoster.FileOmCom.requestFileInformation(FileOmCom.java:222)
    at jd.plugins.hoster.FileOmCom.handleFree(FileOmCom.java:1379)
    at jd.plugins.PluginForHost.handle(PluginForHost.java:814)
    at jd.controlling.downloadcontroller.SingleDownloadController.download(SingleDownloadController.java:387)
    at jd.controlling.downloadcontroller.SingleDownloadController.run(SingleDownloadController.java:502)
Caused by: java.net.ConnectException: Connection refused: connect
    at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at org.appwork.utils.net.httpconnection.HTTPProxyHTTPConnectionImpl.connect(HTTPProxyHTTPConnectionImpl.java:65)
    ... 11 more

--ID:669TS:1408361781506-18/08/14 13.36.21 -  [jd.controlling.downloadcontroller.SingleDownloadController(download)] -> Exception: 
--ID:669TS:1408361781506-18/08/14 13.36.21 -  [] -> java.lang.NullPointerException
    at jd.plugins.hoster.FileOmCom.getPage(FileOmCom.java:1255)
    at jd.plugins.hoster.FileOmCom.requestFileInformation(FileOmCom.java:222)
    at jd.plugins.hoster.FileOmCom.handleFree(FileOmCom.java:1379)
    at jd.plugins.PluginForHost.handle(PluginForHost.java:814)
    at jd.controlling.downloadcontroller.SingleDownloadController.download(SingleDownloadController.java:387)
    at jd.controlling.downloadcontroller.SingleDownloadController.run(SingleDownloadController.java:502)

------------------------Thread: 578:fileom.com-----------------------
--ID:578TS:1408361781508-18/08/14 13.36.21 -  [jd.controlling.downloadcontroller.DownloadWatchDog(handleReturnState)] -> Plugin Defect.1
--ID:578TS:1408361781508-18/08/14 13.36.21 -  [] -> java.lang.Exception: Created Plugin_defect linkresult
    at jd.controlling.downloadcontroller.DownloadLinkCandidateResult.<init>(DownloadLinkCandidateResult.java:124)
    at jd.controlling.downloadcontroller.DownloadWatchDog.handleReturnState(DownloadWatchDog.java:2503)
    at jd.controlling.downloadcontroller.DownloadWatchDog.access$1500(DownloadWatchDog.java:160)
    at jd.controlling.downloadcontroller.DownloadWatchDog$25.execute(DownloadWatchDog.java:2201)
    at jd.controlling.downloadcontroller.DownloadWatchDog$32.processJobs(DownloadWatchDog.java:2839)
    at jd.controlling.downloadcontroller.DownloadWatchDog$32.run(DownloadWatchDog.java:3177)

care of jdlog://0370313173041

NPE caused by socket issue in proxy socket connection, I would call this not a plugin issue myself!
How should we record reports and generate statserv issue tickets about this?

I'm of the opinion we should treat it in the same lines of thirteenth thing when ssl cert/handshake isn't of what we expect. These users/stats should not influence over all statistics for given hoster/object. Effectively its false positive, the only time it wouldn't be would be a issue within our proxy handling (core), though our proxy implementation rarely causes issue and when it does end users would report that issue more so than statserv (finding proxy issues within statserv tickets is a needle in haystack, as its not identified as proxy issue.).

ninth thing
when failures occur (server down time | bugs | <+coalado> the evaluator has been offline), we shouldn't plot those time fields with 0 stats, it should either take a average or stop plotting and recommence plotting when failure has been resolved. This removes issue when averages are damaged due to incorrect values (0), or gives the reader a false positive connotation.

tenth thing
logs dates in statserv tickets are nearly always wrong, even when you've requested a new log! Log dates generally shows up as 'Wed Jun 25'

eleventh thing
statserv seems to keep updating tickets of plugins that have been deleted.

for example I saw unrestricti been updated
https://svn.jdownloader.org/issues/22902 https://svn.jdownloader.org/issues/22900 ~
it was deleted 10+ days earlier https://svn.jdownloader.org/projects/jd/repository/revisions/28896

twelfth thing
reduce the amount of space developer has to download and we have to store on servers. when issue is reported only include update/startup issues and the logs from that plugin, exclude the other plugin and extension/addon logs. This should eliminate the need to download large amount of data for non related objects.

thirteenth thing
track ssl handshake/cert information as general collection of stats. when a issue is lodged by statserv we should discount any reports from users with non standard certs/handshake info. Report back to the user that there network (wan isp or lan gateway) is intercepting secure socket layer connections, in which has most likely caused given issue. This means less false positives with statserv tickets and less interference in statistics

fourteenth thing
currently when exceptions are thrown outside of hoster and decrypter classes it causes epic reliability issues (For example see the SolveMedia example below). My recommendation would be to checksum the offending class (client side), in the report to statserv it should then compare current checksum that statserv has. This works best in my opinion, as reports can not be generated by users with custom classes and classes which are out of date (user hasn't installed core updates, or build block has occurred, or update servers are disabled). If hash doesn't match, the user report and stats are dropped! This should work more effectively vs standard check of $Revision$ at plugin or at the core, as hash would prevent reports from people who have changed source (revisions could still match!). It wont help when underlying issues are caused indirectly for example (re: 'jiaz' user has custom browser class and returns empty response and subsequent null values created and exceptions are thrown because of it.), but to me this would be caused next to never.

java.lang.Exception: SolveMedia Module fails
java.lang.Exception: SolveMedia Module fails

Line 164:                        if (i + 1 != retry) {
Line 165:                            continue;
Line 166:                        }
Line 167:    >>                    throw new Exception("SolveMedia Module fails");


Also available in: Atom PDF