PAW not working after network changes

User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

PAW not working after network changes

Post by gtonkin » Fri Sep 14, 2018 3:50 pm

I have a Win 2016 server running under Hyper-V. I moved the disk image from one host to another which also required new Network Adapters as the virtual switches are different. There are 2 adapters currently bound but only one will ever be active.

PAW was working just fine but now after starting the Admin tool to change the IP addresses, I cannot validate the Application Server Gateway:
PAW_20180914_1736.PNG
PAW_20180914_1736.PNG (33.92 KiB) Viewed 1105 times

My ethernet address is: 192.168.0.30 and the HNS is 172.30.80.1
I can login in to TM1Web, TM1, PAX etc. (Obviously cannot access the editor to change elements/members)
I cannot connect to PAW as it is probably not configured correctly yet.
All PAW processes other than Initialization say Running, Initialization says Completed.

I have tried the following so far:
-Numerous restarts and curses
-Changed the Interface Indexes for the new adapters to be lower than those of the HNS adapter
-Ran NETSTAT -a to see what ports are listening (listing what I think is relevant):
172.30.80.1 ->53, 139
0.0.0.0 -> 5010, 5498, 5895, 5898, 5985, 7999, 9510
I am not seeing anything listening on port 80 yet - expecting this for the PAW login/form

Any suggestions would be very welcome - thanks in advance.

dr.nybble
Posts: 116
Joined: Wed Aug 17, 2011 3:51 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: Excel 2007

Re: PAW not working after network changes

Post by dr.nybble » Mon Sep 17, 2018 1:22 pm

404 means you are talking to the application server, but the /pmhub/pm URL is returning 404.

Make sure PMHub is running. Or if you don't care about PMHub you can ignore it.

You also do not see anything bound to port 80 through the Docker NAT networking with netstat.

Try running scripts/launch.ps1 to connect directly to the PA Gateway container.

User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: PAW not working after network changes

Post by gtonkin » Mon Sep 17, 2018 2:50 pm

Thanks for the response Dr Nybble.
.\scripts\launch.ps1 opens a tab on the browser - http://172.30.88.232/login?cburl=%2F but returns error 500.

I configured config\paw.ps1 with the IP Address the VM is using and tried to access the PAW login from the host machine (i.e. not same as where PAW is running due to the NAT issue).
I get this error now from on http://192.168.43.33/login/form - "Path not exist"
Going to http://192.168.43.33 redirects to http://192.168.43.33/login?cburl=%2F and gives:

Code: Select all

PA-GATEWAY Error: An unexpected error has occurred. See the server logs for more information
Logs view view the admin tool, for Gateway process shows nothing adverse httpd.pid and id.
In the logs\pa-gateway, the *error* log has issues with the FQDN:

Code: Select all

AH00558: httpd.exe: Could not reliably determine the server's fully qualified domain name, using fe80::9d51:833:f4ed:4927. Set the 'ServerName' directive globally to suppress this message
I have used the IP Address everywhere to try avoid this but not sure if the above is a red herring.

Will keep reviewing and reading.

Please let me know if there is anything further I can try or information I can supply.

dr.nybble
Posts: 116
Joined: Wed Aug 17, 2011 3:51 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: Excel 2007

Re: PAW not working after network changes

Post by dr.nybble » Mon Sep 17, 2018 3:34 pm

Look at the wa-proxy logs to see what is throwing the 500 error

The login/form endpoint is not used any more -- all authentication is now handled by the wa-proxy container

User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: PAW not working after network changes

Post by gtonkin » Mon Sep 17, 2018 6:11 pm

Thanks Dr Nubble - the wa-proxy log is below-It contians the original IP Address (192.168.11.113), not the current 192.168.43.33.

Code: Select all

2018-09-17T17:52:51.645Z - error: PA-GATEWAY Error: An unexpected error has occurred

See the server logs for more information 
{ statusCode: 500,
  message: 'PA-GATEWAY Error: An unexpected error has occurred',
  error: 'Error: Failed TM1 login request: Error: connect ETIMEDOUT 192.168.11.113:5010',
  stack: 
   [ 'Error: Failed TM1 login request: Error: connect ETIMEDOUT 192.168.11.113:5010',
     '    at C:\\wa-proxy\\auth\\tm1-strategy.js:283:9',
     '    at tryCatcher (C:\\wa-proxy\\node_modules\\bluebird\\js\\release\\util.js:16:23)',
     '    at Promise._settlePromiseFromHandler (C:\\wa-proxy\\node_modules\\bluebird\\js\\release\\promise.js:512:31)',
     '    at Promise._settlePromise (C:\\wa-proxy\\node_modules\\bluebird\\js\\release\\promise.js:569:18)',
     '    at Promise._settlePromise0 (C:\\wa-proxy\\node_modules\\bluebird\\js\\release\\promise.js:614:10)',
     '    at Promise._settlePromises (C:\\wa-proxy\\node_modules\\bluebird\\js\\release\\promise.js:689:18)',
     '    at Async._drainQueue (C:\\wa-proxy\\node_modules\\bluebird\\js\\release\\async.js:133:16)',
     '    at Async._drainQueues (C:\\wa-proxy\\node_modules\\bluebird\\js\\release\\async.js:143:10)',
     '    at Immediate.Async.drainQueues (C:\\wa-proxy\\node_modules\\bluebird\\js\\release\\async.js:17:14)',
     '    at runCallback (timers.js:574:20)',
     '    at tryOnImmediate (timers.js:554:5)',
     '    at processImmediate [as _immediateCallback] (timers.js:533:5)' ],
  target: 'https://192.168.11.113:5010/api/v1/ActiveUser',
  request: 
   { method: 'GET',
     url: '/login?cburl=%2F',
     remoteIp: '::ffff:172.30.82.220',
     headers: 
      { host: '172.30.82.220',
        accept: 'text/html, application/xhtml+xml, image/jxr, */*',
        referer: 'http://172.30.82.220/',
        'accept-language': 'en-US',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko',
        'accept-encoding': 'gzip, deflate',
        dnt: '1',
        cookie: 'paSession=eyJpZCI6IkdyZjhBZlNNdHh3SE1hTE84bkVHYzFaekMzclRvNzNmIiwiY3JlYXRlZCI6MTUzNzIwNjc1MH0=',
        'x-forwarded-proto': 'http',
        'x-forwarded-for': '172.30.80.1',
        'x-forwarded-host': '172.30.82.220',
        'x-forwarded-server': 'pa-gateway',
        connection: 'Keep-Alive',
        'ba-watson-gateway': 'location=http://wa-proxy:1331;',
        'ba-sso-authenticity': 'blank',
        'ba-sso-user': 'uid=;mail=;cn=Anonymous;loginId=anonymous;loginIdSource=;sessionUUID=Grf8AfSMtxwHMaLO8nEGc1ZzC3rTo73f;',
        'ba-cam-passport': '',
        'ba-scx-subscription': 'tid=null;userid=null;accountid=null;partid=anonymous;',
        'ba-process': 'enabled=false;',
        'ba-share': 'enabled=false;',
        'ba-admin': 'enabled=false;',
        'ba-modeling': 'enabled=false;',
        'ba-social': 'enabled=false;',
        'ba-common_save': 'explicit=false;',
        'ba-graduatedui': 'enabled=false;',
        'ba-welcome': 'largeUpload=disabled;multi-user=enabled;enableWebsheets=true;customFolders=enabled;globalAsk=false;newSamples=false;useCollect=true;smallBanner=true;isBIBannerEnabled=false;isWABannerEnabled=false;',
        'ba-gemini': 'profile=prod;dimensionEditor=enabled;dimensionEditorNewTab=enabled;alternateHierarchy=enabled;allowDynamicSetToStaticInSetEditorWidget=enabled;uploadDataset=disabled;supportMDXInWebsheets=enabled;supportSelectHierarchies=enabled;',
        'ba-tm1servers': 'prod=%5B%7B%22host%22%3A%22192.168.11.113%22%2C%22isSSL%22%3Atrue%2C%22port%22%3A5898%2C%22tcpPort%22%3A5498%2C%22servers%22%3A%5B%5D%7D%5D;',
        'ba-cam-servlet': 'url=http%3A%2F%2Fibmcognoshost%3A9300%2Fp2pd%2Fservlet%2Fdispatch;' } },
  worker: 6 }
My GO_New_Stores model is running and I can log in via Architect. Does not contain any IP Addresses or hard-coded names:

Code: Select all

[TM1S]
DistributedPlanningOutputDir=tunit
PersistentFeeders=T
ForceReevaluationOfFeedersForFedCellsOnDataChange=T
DataBaseDirectory=datafiles
LoggingDirectory=logfiles
ServerName=GO_New_Stores
PortNumber=45557
AdminHost=
Language=eng
Protocol=tcp
AllowSeparateNandCRules = T
NetworkFrame=   
SaveTime=   
DownTime=   
RuleTraceOn=   
RunningInBackground=T
EnableNewHierarchyCreation=T
HTTPPortNumber=5010
EnableTIDebugging=T
UseSSL=T
IntegratedSecurityMode=1
I am not sure where PAW is getting the 192.168.11.113 address from.

Please could you review and let me know if there is something else I should check/change. Thank you.

dr.nybble
Posts: 116
Joined: Wed Aug 17, 2011 3:51 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: Excel 2007

Re: PAW not working after network changes

Post by dr.nybble » Mon Sep 17, 2018 9:02 pm

Verify that the config/paw.ps1 file contains the correct values (if you clicked "Update" in the Administration Tool it should).

If so, run scripts/paw.ps1; this should cause the bss-init service to run and update the user profile with the correct URLs.

User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: PAW not working after network changes

Post by gtonkin » Tue Sep 18, 2018 9:32 am

Hi Dr Nybble, think we are getting closer.

After executing paw.ps1 and starting the Admin tool, I am able to validate and update again.
When checking services however, Gateway shows as Exited, no logs found for Gateway (and in the PA-Gateway folder)
Restarting the server causing the Admin tool to exit with an exception: Execution failed with exit code 1

I have tried rebooting after a successful update but starting the Admin tool and checking services always shows that the Gateway service has exited.

What can you recommend?

Thank you.

Edit: tried to start PA-Gateway manually via Docker start PA-Gateway and received:
Error response from daemon: failed to create endpoint pa-gateway on network nat: HNS failed with error : The object already exists.
Error: failed to start containers: pa-gateway

Edit 2: ran 'docker system prune' and restarted the admintool - may have broken the PA-Gateway as it just says Created and status is ? - starting it crashes the admintool with the error above.

dr.nybble
Posts: 116
Joined: Wed Aug 17, 2011 3:51 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: Excel 2007

Re: PAW not working after network changes

Post by dr.nybble » Tue Sep 18, 2018 3:35 pm

The joys of Docker on Windows!

Try to use the Microsoft-supplied tool to clean up the networking:

https://github.com/MicrosoftDocs/Virtua ... Networking

User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: PAW not working after network changes

Post by gtonkin » Wed Sep 19, 2018 10:15 am

Definitely lots of fun!

No joy yet however - I have tried the script and run it with the cleanup.
At this stage I think I may need to re-install - my PA-Gateway service will just not start and seems to crash the Admin Tool each time.
Going to give this a break for a day or two and see if a fresh mind sorts it.

dr.nybble
Posts: 116
Joined: Wed Aug 17, 2011 3:51 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: Excel 2007

Re: PAW not working after network changes

Post by dr.nybble » Mon Sep 24, 2018 3:32 pm

I have had some success with the Microsoft script with the -ForceDeleteAllSwitches option. Hope Microsoft fixes whatever the underlying issue is that causes the networking to get in a messed up state.

User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: PAW not working after network changes

Post by gtonkin » Mon Sep 24, 2018 3:38 pm

Thanks Dr Nybble, will try this in October. Had to take a week’s leave to get over the anguish it has caused. :lol:

User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: PAW not working after network changes

Post by gtonkin » Thu Oct 04, 2018 10:01 am

The -ForceDeleteAllSwitches option did not work, if anything I seem to be worse off now. I think it is truly broken now.
PAW_20181004_1047.PNG
PAW_20181004_1047.PNG (46.82 KiB) Viewed 569 times
I have tried removing docker completely, re-installing and re-installing the Docker images. Somewhere there still seems to be settings or configuration that I need to remove, re-initialise.

Is there a way to start from fresh?

dr.nybble
Posts: 116
Joined: Wed Aug 17, 2011 3:51 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: Excel 2007

Re: PAW not working after network changes

Post by dr.nybble » Thu Oct 04, 2018 1:39 pm


User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: PAW not working after network changes

Post by gtonkin » Thu Oct 04, 2018 2:34 pm

Thanks Dr Nybble, had happened on the zap tool a couple of hours ago and uninstalled, zapped, reinstalled, recreated PAW from images, busy reconfiguring etc.

Will post back once all is done...

User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: PAW not working after network changes

Post by gtonkin » Thu Oct 04, 2018 7:36 pm

After re-installing, everything seems to be up and running except there is no Gateway service??????
PAW_20181004_2135.PNG
PAW_20181004_2135.PNG (43.67 KiB) Viewed 541 times
Docker images does show the image is available.

Is there a way to manually create and trigger the Gateway service - it should happen automatically.

User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: PAW not working after network changes

Post by gtonkin » Thu Oct 04, 2018 8:13 pm

Something definitely not right with PAW on this machine - after all that, my windowsfilter folder is now 70Gb again and cannot start services as there is no space left on my device.

User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: PAW not working after network changes

Post by gtonkin » Sat Oct 06, 2018 9:00 am

Making some progress but the Admintool is having issues starting:

Code: Select all

Removing admintool
Recreating 92bea6393ef1_admintool ... error

ERROR: for 92bea6393ef1_admintool  driver "windowsfilter" failed to remove root filesystem for 92bea6393ef1e623c547668c4
797401ed11357d8b3fe4f4333db4200b6a40c51: rename C:\ProgramData\docker\windowsfilter\92bea6393ef1e623c547668c4797401ed113
57d8b3fe4f4333db4200b6a40c51 C:\ProgramData\docker\windowsfilter\92bea6393ef1e623c547668c4797401ed11357d8b3fe4f4333db420
0b6a40c51-removing: Access is denied.

ERROR: for admintool  driver "windowsfilter" failed to remove root filesystem for 92bea6393ef1e623c547668c4797401ed11357
d8b3fe4f4333db4200b6a40c51: rename C:\ProgramData\docker\windowsfilter\92bea6393ef1e623c547668c4797401ed11357d8b3fe4f433
3db4200b6a40c51 C:\ProgramData\docker\windowsfilter\92bea6393ef1e623c547668c4797401ed11357d8b3fe4f4333db4200b6a40c51-rem
oving: Access is denied.
ERROR: Encountered errors while bringing up the project.
Using process explorer, I can see that System as a handle on all the windowsfilter folders but not sure what else could be locking or why.
Any suggestions?

dr.nybble
Posts: 116
Joined: Wed Aug 17, 2011 3:51 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: Excel 2007

Re: PAW not working after network changes

Post by dr.nybble » Tue Oct 09, 2018 1:05 pm

There is a logged defect for Docker that you may be hitting. It is related to a daemon setting, 'shutdown-timeout'.

https://success.docker.com/api/articles ... ainers/pdf

See if a reboot fixes it (it would unmount the VHD).

Generally, the better performing your Windows 2016 system the fewer issues you see. How many cores does your system have?

User avatar
gtonkin
MVP
Posts: 638
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: PAW not working after network changes

Post by gtonkin » Tue Oct 09, 2018 1:47 pm

Thank you for the link - will have a read through it shortly.

My VM has 12 processors assigned and 16Gb RAM, 10.8Gb currently in use.

When running >docker ps, I am expecting to see PA_Gateway but do not see it. It somehow did not get added. I did try the upgrade/install option again when starting the Admin Tool but without success. All other services seem to eventually enter a running state, other than Initialization.

Code: Select all

CONTAINER ID        IMAGE                                                                       COMMAND                   CREATED             STATUS              PORTS                    NAMES
5e482ca0cf7c        pa-docker:5000/planninganalytics/admintool:1.0.72-windowsservercore         "powershell -Comma..."    2 days ago          Up 2 days           0.0.0.0:8888->8888/tcp   admintool
5e4995e4afd8        pa-docker:5000/planninganalytics/wa-proxy:1.0.247-windowsservercore         "powershell -Comma..."    2 days ago          Up 2 days           1338/tcp                 wa-proxy
c2e915bda48f        pa-docker:5000/planninganalytics/share-platform:1.0.55-windowsservercore    "powershell -Comma..."    2 days ago          Up 2 days                                    share-platform
9bfb9bc7cdf5        pa-docker:5000/planninganalytics/welcome:1.0.185-windowsservercore          "powershell -Comma..."    2 days ago          Up 2 days                                    welcome
854e7db2ec82        pa-docker:5000/planninganalytics/prism-app:1.0.3022.2-windowsservercore     "powershell -c C:\\..."   2 days ago          Up 2 days                                    prism-app
9666319ce8f4        pa-docker:5000/planninganalytics/prism-platform:1.0.856-windowsservercore   "powershell -Comma..."    2 days ago          Up 2 days                                    prism-platform
14dc6ff2fec3        pa-docker:5000/planninganalytics/share-proxy:1.0.119-windowsservercore      "powershell -Comma..."    2 days ago          Up 2 days                                    share-proxy
16a342f3ed85        pa-docker:5000/planninganalytics/share-app:1.0.24-windowsservercore         "powershell -Comma..."    2 days ago          Up 2 days                                    share-app
d377f88f1077        pa-docker:5000/planninganalytics/redis:2.8.2402-windowsservercore           "powershell -Comma..."    2 days ago          Up 2 days           6379/tcp                 redis
88eefad393cf        pa-docker:5000/planninganalytics/mongo:3.2-windowsservercore                "mongod --smallfil..."    2 days ago          Up 2 days           27017/tcp                mongo
15435c156593        pa-docker:5000/planninganalytics/bss-init:1.0.80-windowsservercore          "powershell -Comma..."    2 days ago          Up 2 days                                    bss-init
fa1a3653f86f        pa-docker:5000/planninganalytics/couchdb:2.0.0.1-windowsservercore          "powershell -Comma..."    2 days ago          Up 2 days           5984/tcp                 couchdb
3ae9e16d30b6        pa-docker:5000/planninganalytics/couchdb_init:1.0.213-windowsservercore     "powershell -Comma..."    2 days ago          Up 2 days                                    couchdb_init
1ff9c9b6a58f        pa-docker:5000/planninganalytics/neo-idviz:1.0.232-windowsservercore        "powershell -Comma..."    2 days ago          Up 2 days           9060/tcp                 neo-idviz
d268a1532a21        pa-docker:5000/planninganalytics/bss:1.0.269-windowsservercore              "powershell -Comma..."    2 days ago          Up 2 days           8082/tcp                 bss
e02003c4c542        pa-docker:5000/planninganalytics/user-admin:1.0.112-windowsservercore       "powershell -Comma..."    2 days ago          Up 2 days           3333/tcp                 user-admin
168487d24fc2        pa-docker:5000/planninganalytics/cdn3p:1.0.61-windowsservercore             "powershell -Comma..."    4 days ago          Up 2 days                                    cdn3p
be707525a55f        pa-docker:5000/planninganalytics/prism-proxy:1.0.3819.1-windowsservercore   "powershell -Comma..."    4 days ago          Up 2 days                                    prism-proxy
5547572128ad        pa-docker:5000/planninganalytics/glass:1.0.336-windowsservercore            "powershell -Comma..."    4 days ago          Up 2 days                                    glass
6d32646f3fb5        pa-docker:5000/planninganalytics/neo-provision:1.0.227-windowsservercore    "powershell -Comma..."    4 days ago          Up 2 days           8083/tcp                 neo-provision

dr.nybble
Posts: 116
Joined: Wed Aug 17, 2011 3:51 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: Excel 2007

Re: PAW not working after network changes

Post by dr.nybble » Tue Oct 09, 2018 2:11 pm

Run:
scripts/paw.ps1

You may see all containers (including stopped ones) with docker ps -a

The initialization container will exit when it has done its job.

Post Reply