Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

trickster does not suport /api/v1/series on multi-origin #27

Closed
roidelapluie opened this issue Apr 9, 2018 · 12 comments
Closed

trickster does not suport /api/v1/series on multi-origin #27

roidelapluie opened this issue Apr 9, 2018 · 12 comments

Comments

@roidelapluie
Copy link
Contributor

Apr 09 16:43:34 grafana01 trickster[18199]: 2018/04/09 16:43:34 http: panic serving 127.0.0.1:34936: runtime error: invalid memory address
or nil pointer dereference
Apr 09 16:43:34 grafana01 trickster[18199]: goroutine 4078 [running]:
Apr 09 16:43:34 grafana01 trickster[18199]: net/http.(*conn).serve.func1(0xc425de3040)
Apr 09 16:43:34 grafana01 trickster[18199]: /usr/local/go/src/net/http/server.go:1697 +0xd0
Apr 09 16:43:34 grafana01 trickster[18199]: panic(0x777940, 0xa08940)
Apr 09 16:43:34 grafana01 trickster[18199]: /usr/local/go/src/runtime/panic.go:491 +0x283
Apr 09 16:43:34 grafana01 trickster[18199]: main.(*TricksterHandler).promFullProxyHandler(0xc420054780, 0x9dcc00, 0xc428f8f950, 0xc434c8170
0)
Apr 09 16:43:34 grafana01 trickster[18199]: /repository/workspace/modules/com.github.comcast.trickster/master/release/src/github.com/comcas
t/trickster/handlers.go:74 +0x3f7
Apr 09 16:43:34 grafana01 trickster[18199]: main.(*TricksterHandler).(main.promFullProxyHandler)-fm(0x9dcc00, 0xc428f8f950, 0xc434c81700)
Apr 09 16:43:34 grafana01 trickster[18199]: /repository/workspace/modules/com.github.comcast.trickster/master/release/src/github.com/comcas
t/trickster/main.go:67 +0x48
Apr 09 16:43:34 grafana01 trickster[18199]: net/http.HandlerFunc.ServeHTTP(0xc42011a710, 0x9dcc00, 0xc428f8f950, 0xc434c81700)
Apr 09 16:43:34 grafana01 trickster[18199]: /usr/local/go/src/net/http/server.go:1918 +0x44
Apr 09 16:43:34 grafana01 trickster[18199]: github.com/comcast/trickster/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc4200f96c0, 0x
9dcc00, 0xc428f8f950, 0xc434c81700)
Apr 09 16:43:34 grafana01 trickster[18199]: /repository/workspace/modules/com.github.comcast.trickster/master/release/src/github.com/comcas
t/trickster/vendor/github.com/gorilla/mux/mux.go:159 +0xed
Apr 09 16:43:34 grafana01 trickster[18199]: github.com/comcast/trickster/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1(0x9d
d140, 0xc4249fd960, 0xc423592100)
Apr 09 16:43:34 grafana01 trickster[18199]: /repository/workspace/modules/com.github.comcast.trickster/master/release/src/github.com/comcas
t/trickster/vendor/github.com/gorilla/handlers/compress.go:146 +0x462
Apr 09 16:43:34 grafana01 trickster[18199]: net/http.HandlerFunc.ServeHTTP(0xc42011c920, 0x9dd140, 0xc4249fd960, 0xc423592100)
Apr 09 16:43:34 grafana01 trickster[18199]: /usr/local/go/src/net/http/server.go:1918 +0x44
Apr 09 16:43:34 grafana01 trickster[18199]: net/http.serverHandler.ServeHTTP(0xc420073860, 0x9dd140, 0xc4249fd960, 0xc423592100)
Apr 09 16:43:34 grafana01 trickster[18199]: /usr/local/go/src/net/http/server.go:2619 +0xb4
Apr 09 16:43:34 grafana01 trickster[18199]: net/http.(*conn).serve(0xc425de3040, 0x9dd980, 0xc422cfdec0)
Apr 09 16:43:34 grafana01 trickster[18199]: /usr/local/go/src/net/http/server.go:1801 +0x71d
Apr 09 16:43:34 grafana01 trickster[18199]: created by net/http.(*Server).Serve
Apr 09 16:43:34 grafana01 trickster[18199]: /usr/local/go/src/net/http/server.go:2720 +0x288
@roidelapluie
Copy link
Contributor Author

Note: cache_type = 'filesystem'

@roidelapluie
Copy link
Contributor Author

roidelapluie commented Apr 9, 2018 •
edited
Loading

Jest before: time=2018-04-09T14:43:34.491787239Z app=trickster caller=trickster/handlers.go:225 level=error
event="error downloading url" url="http://prometheus:9090//foo2/api/v1/series?end=1523284955&match%5B%5D=ifHCOutOctets&start=1523277755" detail="Get http://prometheus:9090/foo2/api/v1/series?end=1523284955&match%5B%5D=ifHCOutOctets&start=1523277755: dial tcp 155.45.169.233:9090: i/o timeout"

@roidelapluie
Copy link
Contributor Author

Looks like origin_url = 'http://10.19.2.37:9090' is ignored in my config

@crandles
Copy link
Contributor

crandles commented Apr 9, 2018

Can you share your config?

@roidelapluie
Copy link
Contributor Author 

[main]                                                                                                                                    
[proxy_server]                                                                                                                            
    listen_port = 9090                                                                                                                    
    listen_address = "127.0.0.1"                                                                                                          
[cache]                                                                                                                                   
    cache_type = 'filesystem'                                                                                                             
    record_ttl_secs = 21600                                                                                                               
        [cache.filesystem]                                                                                                                
            cache_path = '/srv/grafana/trickster'                                                                                         
[origins]                                                                                                                                 
    [origins.foo1]                                                                                                                  
        origin_url = 'http://10.19.2.37:9090'                                                                                           
        api_path = '/api/v1'                                                                                                              
        default_step = 300                                                                                                                
        max_value_age_secs = 86400                                                                                                        
    [origins.foo2]                                                                                                                  
        origin_url = 'http://10.99.2.12:9090'                                                                                          
        api_path = '/api/v1'                                                                                                              
        default_step = 300                                                                                                                
        max_value_age_secs = 86400                                                                                                        
[metrics]                                                                                                                                 
    listen_port = 16234                                                                                                                   
    listen_address = "127.0.0.1"                                                                                                          
[logging]                                                                                                                                 
    log_level = 'info'

@roidelapluie
Copy link
Contributor Author

I have 2 patches installed: #26 and #24

@roidelapluie roidelapluie changed the title promFullProxyHandler Panic trickster does not suport /api/v1/series on multi-origin Apr 9, 2018
@roidelapluie
Copy link
Contributor Author

roidelapluie commented Apr 9, 2018 •
edited
Loading

The actual issue is: trickster does not suport /api/v1/series on multi-origin path-based #27

@jranson
Copy link
Member

jranson commented Apr 9, 2018

@roidelapluie The panic appears to be a bug we will take a look at (i'll re-update the issue topic), but your overall situation (not picking up your configured origins) looks to be an issue with how you are configuring multi-origin in Trickster and in Grafana. Short answer: user error. You did not configure a default origin ([origins.default]), so when you do not provide a host-header-based, path-based or parameter-based origin moniker in your Grafana config, Trickster will choose the default origin of http://prometheus:9090. You should be able to update your Grafana Data Sources configs by appending foo1 or foo2 to the paths (e.g., http://<your_trickster_endpoint:9090/foo1) as described in [https://github.com/Comcast/trickster/blob/master/docs/multi-origin.md]. The "path-based" section (complete with a screen shot of how to configure Grafana) should point you in the right direction.

@roidelapluie
Copy link
Contributor Author

roidelapluie commented Apr 9, 2018 •
edited
Loading

@jranson The error is that we do not proxy /api/v1/series when using path based multi origin. The panic is because we ignore proxy errors.

@jranson
Copy link
Member

jranson commented Apr 9, 2018

Ah, so this is specific to /series (but not /query and /query_range)? Very interesting! Will take a look today!

@roidelapluie
Copy link
Contributor Author

I can not get this to work for now.

@jranson
Copy link
Member

jranson commented Apr 10, 2018

@roidelapluie I've found the root cause and will issue a fix shortly.

@jranson jranson reopened this Apr 10, 2018
@jranson jranson mentioned this issue Apr 10, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@crandles @roidelapluie @jranson